Where Do Large Learning Rates Lead Us?

TL;DR

This paper empirically investigates how the choice of large initial learning rates affects neural network training, revealing that only a narrow optimal range leads to high-quality, well-generalized minima with focused features.

Contribution

It identifies the optimal initial learning rate range for training, linking it to basin geometry and feature sparsity, advancing understanding of learning rate effects.

Findings

Optimal initial LRs are narrowly above the convergence threshold.

Using optimal LRs leads to high-quality minima with focused features.

Too small or too large LRs result in poor generalization and unstable minima.

Abstract

It is generally accepted that starting neural networks training with large learning rates (LRs) improves generalization. Following a line of research devoted to understanding this effect, we conduct an empirical study in a controlled setting focusing on two questions: 1) how large an initial LR is required for obtaining optimal quality, and 2) what are the key differences between models trained with different LRs? We discover that only a narrow range of initial LRs slightly above the convergence threshold lead to optimal results after fine-tuning with a small LR or weight averaging. By studying the local geometry of reached minima, we observe that using LRs from this optimal range allows for the optimization to locate a basin that only contains high-quality minima. Additionally, we show that these initial LRs result in a sparse set of learned features, with a clear focus on those most relevant for the task. In contrast, starting training with too small LRs leads to unstable minima and attempts to learn all features simultaneously, resulting in poor generalization. Conversely, using initial LRs that are too large fails to detect a basin with good solutions and extract meaningful patterns from the data.