Improved Scaling Laws in Linear Regression via Data Reuse

TL;DR

This paper demonstrates that data reuse in linear regression with multi-pass SGD can improve test error bounds and scaling laws, especially when data is limited, by leveraging spectral properties of data covariance and prior distributions.

Contribution

The paper derives sharp test error bounds showing how data reuse via multi-pass SGD improves scaling laws in linear regression under spectral assumptions.

Findings

Multi-pass SGD achieves lower test error bounds than one-pass SGD.

Data reuse improves scaling laws in data-constrained regimes.

Numerical simulations verify theoretical predictions.

Abstract

Neural scaling laws suggest that the test error of large language models trained online decreases polynomially as the model size and data size increase. However, such scaling can be unsustainable when running out of new data. In this work, we show that data reuse can improve existing scaling laws in linear regression. Specifically, we derive sharp test error bounds on $M$-dimensional linear models trained by multi-pass stochastic gradient descent (multi-pass SGD) on $N$ data with sketched features. Assuming that the data covariance has a power-law spectrum of degree $a$, and that the true parameter follows a prior with an aligned power-law spectrum of degree $b-a$ (with $a > b > 1$), we show that multi-pass SGD achieves a test error of $\Theta(M^{1-b} + L^{(1-b)/a})$, where $L \lesssim N^{a/b}$ is the number of iterations. In the same setting, one-pass SGD only attains a test error of $\Theta(M^{1-b} + N^{(1-b)/a})$ (see e.g., Lin et al., 2024). This suggests an improved scaling law via data reuse (i.e., choosing $L>N$) in data-constrained regimes. Numerical simulations are also provided to verify our theoretical findings.