The Break-Even Point on Optimization Trajectories of Deep Neural Networks

TL;DR

This paper investigates how the early phase of training deep neural networks with SGD influences their final performance, highlighting the importance of the break-even point where regularization effects emerge.

Contribution

It introduces the concept of the break-even point in optimization trajectories and demonstrates how initial hyperparameters affect the loss surface and gradient noise.

Findings

Large initial learning rates reduce gradient variance.

Early phase training impacts the conditioning of the loss surface.

Low learning rates lead to poor loss surface conditioning.

Abstract

The early phase of training of deep neural networks is critical for their final performance. In this work, we study how the hyperparameters of stochastic gradient descent (SGD) used in the early phase of training affect the rest of the optimization trajectory. We argue for the existence of the "break-even" point on this trajectory, beyond which the curvature of the loss surface and noise in the gradient are implicitly regularized by SGD. In particular, we demonstrate on multiple classification tasks that using a large learning rate in the initial phase of training reduces the variance of the gradient, and improves the conditioning of the covariance of gradients. These effects are beneficial from the optimization perspective and become visible after the break-even point. Complementing prior work, we also show that using a low learning rate results in bad conditioning of the loss surface even for a neural network with batch normalization layers. In short, our work shows that key properties of the loss surface are strongly influenced by SGD in the early phase of training. We argue that studying the impact of the identified effects on generalization is a promising future direction.