Time Transfer: On Optimal Learning Rate and Batch Size In The Infinite Data Limit

TL;DR

This paper investigates the optimal scaling of learning rate and batch size in large language model pretraining as data size approaches infinity, revealing intricate dependencies on data and model parameters.

Contribution

It uncovers the dependence of optimal learning rate and batch size on pretraining token budget and critical batch size, extending scaling rules to the infinite data limit.

Findings

Optimal $ ext{η}$ and $B$ depend on token budget $T$ and critical batch size $B_ ext{crit}$.

Critical batch size $B_ ext{crit}$ scales proportionally with $T$.

Sensitivity of loss to learning rate decreases with increasing $T$.

Abstract

One of the main challenges in optimal scaling of large language models (LLMs) is the prohibitive cost of hyperparameter tuning, particularly learning rate $\eta$ and batch size $B$. While techniques like $\mu$P (Yang et al., 2022) provide scaling rules for optimal $\eta$ transfer in the infinite model size limit, the optimal scaling behavior in the infinite data size limit remains unknown. We fill in this gap by observing for the first time an intricate dependence of optimal $\eta$ scaling on the pretraining token budget $T$, $B$ and its relation to the critical batch size $B_\mathrm{crit}$, which we measure to evolve as $B_\mathrm{crit} \propto T$. Furthermore, we show that the optimal batch size is positively correlated with $B_\mathrm{crit}$: keeping it fixed becomes suboptimal over time even if learning rate is scaled optimally. Surprisingly, our results demonstrate that the observed optimal $\eta$ and $B$ dynamics are preserved with $\mu$P model scaling, challenging the conventional view of $B_\mathrm{crit}$ dependence solely on loss value. Complementing optimality, we examine the sensitivity of loss to changes in learning rate, where we find the sensitivity to decrease with increase of $T$ and to remain constant with $\mu$P model scaling. We hope our results make the first step towards a unified picture of the joint optimal data and model scaling.