Predictable Scale: Part II, Farseer: A Refined Scaling Law in Large Language Models

TL;DR

Farseer is a refined scaling law for large language models that significantly improves predictive accuracy over previous laws, enabling reliable extrapolation from small-scale experiments to large-scale training.

Contribution

We introduce Farseer, a new scaling law that offers more accurate and generalizable predictions of LLM performance across scales, surpassing prior models like Chinchilla's law.

Findings

Farseer reduces extrapolation error by 433%.

It accurately predicts performance across diverse model scales.

Validated with ~1,000 models trained on 3 million GPU hours.

Abstract

Training Large Language Models (LLMs) is prohibitively expensive, creating a critical scaling gap where insights from small-scale experiments often fail to transfer to resource-intensive production systems, thereby hindering efficient innovation. To bridge this, we introduce Farseer, a novel and refined scaling law offering enhanced predictive accuracy across scales. By systematically constructing a model loss surface $L(N,D)$, Farseer achieves a significantly better fit to empirical data than prior laws (e.g., Chinchilla's law). Our methodology yields accurate, robust, and highly generalizable predictions, demonstrating excellent extrapolation capabilities, improving upon Chinchilla's law by reducing extrapolation error by 433\%. This allows for the reliable evaluation of competing training strategies across all $(N,D)$ settings, enabling conclusions from small-scale ablation studies to be confidently extrapolated to predict large-scale performance. Furthermore, Farseer provides new insights into optimal compute allocation, better reflecting the nuanced demands of modern LLM training. To validate our approach, we trained an extensive suite of approximately 1,000 LLMs across diverse scales and configurations, consuming roughly 3 million NVIDIA H100 GPU hours. We are comprehensively open-sourcing all models, data, results, and logs at https://github.com/Farseer-Scaling-Law/Farseer to foster further research.