Problems with Chinchilla Approach 2: Systematic Biases in IsoFLOP Parabola Fits

TL;DR

This paper identifies systematic biases in the widely used Chinchilla Approach 2 for fitting neural scaling laws, proposes improvements, and demonstrates more accurate, unbiased inference methods with practical advantages.

Contribution

It introduces Variable Projection to address biases in Approach 2, offering a more stable, unbiased, and scalable method for neural scaling law fitting.

Findings

Chinchilla Approach 2 introduces biases leading to underallocation and unnecessary compute.

Approach 3 reduces biases but has perceived drawbacks that are addressable.

Variable Projection enables unbiased, well-conditioned inference on all loss surface parameters.

Abstract

Chinchilla Approach 2 is among the most widely used methods for fitting neural scaling laws. Its parabolic approximation introduces systematic biases in compute-optimal allocation estimates, even on noise-free synthetic data. Applied to published Llama 3 IsoFLOP data at open frontier compute scales, these biases imply a parameter underallocation corresponding to 6.5% of the $3.8\times10^{25}$ FLOP training budget and \$1.4M (90% CI: \$412K-\$2.9M) in unnecessary compute at 50% H100 MFU. Simulated multimodal model misallocations show even greater opportunity costs due to higher loss surface asymmetry. Three sources of this error are examined: IsoFLOP sampling grid width (Taylor approximation accuracy), uncentered IsoFLOP sampling, and loss surface asymmetry ($\alpha \neq \beta$). Chinchilla Approach 3 largely eliminates these biases but is often regarded as less data-efficient, numerically unstable, prone to local minima, and harder to implement. Each concern is shown to be unfounded or addressable, especially when the partially linear structure of the objective is exploited via Variable Projection, enabling unbiased inference on all five loss surface parameters through a two-dimensional optimization that is well-conditioned, analytically differentiable, and amenable to dense, or even exhaustive, grid search. It may serve as a more convenient replacement for Approach 2 or a more scalable alternative for adaptations of Approach 3 to richer scaling law formulations. See https://github.com/Open-Athena/vpnls for details and https://openathena.ai/scaling-law-analysis for other results from this study.