A Generalized Spectral Framework to Expain Neural Scaling and Compression Dynamics

TL;DR

This paper introduces a generalized spectral framework that unifies neural learning dynamics and compression phenomena, explaining diverse scaling behaviors through a common mathematical model.

Contribution

It develops a spectral evolution function that generalizes existing theories, connecting learning and compression under a unified framework.

Findings

Recovers lazy and feature-learning theories as special cases

Provides an invariant relation between learning and compression

Unifies diverse neural scaling behaviors

Abstract

Empirical scaling laws describe how test loss and other performance metrics depend on model size, dataset size, and compute. While such laws are consistent within specific regimes, apparently distinct scaling behaviors have been reported for related settings such as model compression. Motivated by recent progress in spectral analyses of neural representations, this paper develops a \emph{generalized spectral framework} that unifies learning dynamics and compression phenomena under a common functional ansatz. We generalize the spectral evolution function from the linear kernel form $g(\lambda t)=\lambda t$ to an asymptotically polynomial function $g(\lambda,t;\beta)$, characterized by an effective spectral--temporal elasticity $\rho(\beta)$. This framework recovers existing lazy and feature-learning theories as special cases and yields an invariant relation between learning and compression