How Does Learning Rate Decay Help Modern Neural Networks?

TL;DR

This paper proposes a new explanation for how learning rate decay improves training of modern neural networks, suggesting it suppresses memorization of noise initially and enhances learning of complex patterns later.

Contribution

It introduces a novel perspective on lrDecay's effectiveness, supported by experiments on synthetic and real datasets, challenging traditional optimization-based explanations.

Findings

Large initial learning rate prevents memorizing noise

Decaying learning rate enhances learning of complex patterns

Later-stage learned patterns are less transferable

Abstract

Learning rate decay (lrDecay) is a \emph{de facto} technique for training modern neural networks. It starts with a large learning rate and then decays it multiple times. It is empirically observed to help both optimization and generalization. Common beliefs in how lrDecay works come from the optimization analysis of (Stochastic) Gradient Descent: 1) an initially large learning rate accelerates training or helps the network escape spurious local minima; 2) decaying the learning rate helps the network converge to a local minimum and avoid oscillation. Despite the popularity of these common beliefs, experiments suggest that they are insufficient in explaining the general effectiveness of lrDecay in training modern neural networks that are deep, wide, and nonconvex. We provide another novel explanation: an initially large learning rate suppresses the network from memorizing noisy data while decaying the learning rate improves the learning of complex patterns. The proposed explanation is validated on a carefully-constructed dataset with tractable pattern complexity. And its implication, that additional patterns learned in later stages of lrDecay are more complex and thus less transferable, is justified in real-world datasets. We believe that this alternative explanation will shed light into the design of better training strategies for modern neural networks.