More Compute Is What You Need

TL;DR

This paper proposes a new scaling law indicating that for transformer models, performance primarily depends on total compute, guiding training strategies towards smaller models with larger datasets for efficiency and larger models when datasets are exhausted.

Contribution

It introduces a unified scaling law that emphasizes compute as the main factor for model performance, challenging traditional size and dataset allocation assumptions.

Findings

Performance depends mainly on total compute, not size or dataset.

Training should focus on smaller models with larger datasets for efficiency.

Scaling model size is key when datasets are exhausted.

Abstract

Large language model pre-training has become increasingly expensive, with most practitioners relying on scaling laws to allocate compute budgets for model size and training tokens, commonly referred to as Compute-Optimal or Chinchilla Optimal. In this paper, we hypothesize a new scaling law that suggests model performance depends mostly on the amount of compute spent for transformer-based models, independent of the specific allocation to model size and dataset size. Using this unified scaling law, we predict that (a) for inference efficiency, training should prioritize smaller model sizes and larger training datasets, and (b) assuming the exhaustion of available web datasets, scaling the model size might be the only way to further improve model performance.