Empirical Laws Governing Deep Learning Performance
Answer: Below are the main empirical laws and regularities the paper identifies as governing deep‑learning performance, with direct citations for every sentence. Neural scaling laws: test loss (final loss) follows pre...
Answer: Below are the main empirical laws and regularities the paper identifies as governing deep‑learning performance, with direct citations for every sentence. Neural scaling laws: test loss (final loss) follows predictable power‑law scaling in compute, dataset size, and parameter count within an architectural family.[:cite[1]{ln=1}], [:cite[2]{ln=4}] Edge of stability / progressive sharpening: during (full‑batch) gradient‑descent training, Hessian sharpness grows (progressive sharpening) and then plateaus near the value 2/η (the edge of stability).[:cite[3]{ln=1}], [:cite[4]{ln=2}] Conservation laws under gradient flow: certain layerwise matrix differences (e.g., WℓWℓ⊤ − Wℓ+1⊤Wℓ+1) are conserved under gradient flow, a pattern explained by continuous symmetries (Noether‑type reasoning).[:cite[5]{ln=1}], [:cite[5]{ln=2}] Neural collapse: in classifiers, final hidden‑layer representations of each class cluster around class means which form a regular simplex at the end of training.[:cite[6]{ln=1}], [:cite[6]{ln=4}] First‑layer/feature alignment (neural feature ansatz): the first‑layer Gram matrix aligns with the average gradient outer product after training (a heuristic that matches experiments).[:cite[6]{ln=7}], [:cite[6]{ln=10}] Joint/proportional scaling limits and disentangling hyperparameters: many useful limits commute or require joint scaling (e.g., data and parameter counts), and hyperparameters (learning rate, batch size, initialization, width/depth) admit limiting descriptions that let them be disentangled from simpler effective dynamics.[:cite[7]{ln=1}], [:cite[9]{ln=1}], [:cite[8]{ln=2}] SGD/optimizer implicit curvature regularization: optimizer hyperparameters (large learning rates, small batch sizes) implicitly regularize loss curvature (Hessian), producing predictable effects on trajectories and learned networks.[:cite[11]{ln=1}], [:cite[10]{ln=1}] Missing / open points the paper emphasizes (i.e., limits of current empirical laws): The origin and exponents of neural scaling power laws (why power laws arise and what sets the exponents) remain not fully explained from first principles.[:cite[13]{ln=1}], [:cite[12]{ln=1}] Quantitative theory for progressive sharpening in realistic nonlinear networks is still incomplete.[:cite[4]{ln=1}] If you want, I can (1) produce a concise one‑page summary with these laws and the exact paragraph citations, or (2) extract the figures/tables referenced (e.g., Figure 3 on scaling laws, Figure 4 on edge of stability) with their captions and citations. Which would you prefer?