Surge Phenomenon in Optimal Learning Rate and Batch Size Scaling

TL;DR

This paper investigates the relationship between optimal learning rates and batch sizes for Adam style optimizers, revealing a surge phenomenon where the optimal learning rate first increases then decreases with batch size, supported by theory and experiments.

Contribution

It introduces a new scaling law for Adam style optimizers showing a surge in optimal learning rate with batch size, supported by theoretical proof and extensive experiments.

Findings

Optimal learning rate first rises then falls with batch size.

The surge peak shifts toward larger batch sizes over training.

Experimental results verify the proposed scaling law.

Abstract

In current deep learning tasks, Adam style optimizers such as Adam, Adagrad, RMSProp, Adafactor, and Lion have been widely used as alternatives to SGD style optimizers. These optimizers typically update model parameters using the sign of gradients, resulting in more stable convergence curves. The learning rate and the batch size are the most critical hyperparameters for optimizers, which require careful tuning to enable effective convergence. Previous research has shown that the optimal learning rate increases linearly or follows similar rules with batch size for SGD style optimizers. However, this conclusion is not applicable to Adam style optimizers. In this paper, we elucidate the connection between optimal learning rates and batch sizes for Adam style optimizers through both theoretical analysis and extensive experiments. First, we raise the scaling law between batch sizes and optimal learning rates in the sign of gradient case, in which we prove that the optimal learning rate first rises and then falls as the batch size increases. Moreover, the peak value of the surge will gradually move toward the larger batch size as training progresses. Second, we conducted experiments on various CV and NLP tasks and verified the correctness of the scaling law.