Revisiting Neural Scaling Laws in Language and Vision

TL;DR

This paper introduces a rigorous methodology for estimating neural scaling laws using extrapolation loss, improving prediction accuracy across various domains and architectures, and provides a new benchmark dataset for future research.

Contribution

It proposes a novel extrapolation-based method for reliably estimating scaling law parameters from learning curves, enhancing the understanding of model scaling benefits.

Findings

Extrapolation loss outperforms previous fitting methods in predicting scaling benefits.

The method applies across diverse architectures and tasks, including image classification and language modeling.

A new benchmark dataset with 90 tasks is released to support further research.

Abstract

The remarkable progress in deep learning in recent years is largely driven by improvements in scale, where bigger models are trained on larger datasets for longer schedules. To predict the benefit of scale empirically, we argue for a more rigorous methodology based on the extrapolation loss, instead of reporting the best-fitting (interpolating) parameters. We then present a recipe for estimating scaling law parameters reliably from learning curves. We demonstrate that it extrapolates more accurately than previous methods in a wide range of architecture families across several domains, including image classification, neural machine translation (NMT) and language modeling, in addition to tasks from the BIG-Bench evaluation benchmark. Finally, we release a benchmark dataset comprising of 90 evaluation tasks to facilitate research in this domain.