Towards Explaining the Regularization Effect of Initial Large Learning Rate in Training Neural Networks

TL;DR

This paper explains why large initial learning rates in neural network training lead to better generalization after annealing, by analyzing pattern learning order and demonstrating effects on image classification.

Contribution

It provides a theoretical and empirical analysis showing large initial learning rates promote better generalization by influencing pattern learning order in neural networks.

Findings

Large initial learning rate improves generalization after annealing.

Small learning rate models memorize easy patterns first, harming generalization.

Adding a patch to images reveals differences in learning dynamics.

Abstract

Stochastic gradient descent with a large initial learning rate is widely used for training modern neural net architectures. Although a small initial learning rate allows for faster training and better test performance initially, the large learning rate achieves better generalization soon after the learning rate is annealed. Towards explaining this phenomenon, we devise a setting in which we can prove that a two layer network trained with large initial learning rate and annealing provably generalizes better than the same network trained with a small learning rate from the start. The key insight in our analysis is that the order of learning different types of patterns is crucial: because the small learning rate model first memorizes easy-to-generalize, hard-to-fit patterns, it generalizes worse on hard-to-generalize, easier-to-fit patterns than its large learning rate counterpart. This concept translates to a larger-scale setting: we demonstrate that one can add a small patch to CIFAR-10 images that is immediately memorizable by a model with small initial learning rate, but ignored by the model with large learning rate until after annealing. Our experiments show that this causes the small learning rate model's accuracy on unmodified images to suffer, as it relies too much on the patch early on.