A Convergence Analysis of Nesterov's Accelerated Gradient Method in Training Deep Linear Neural Networks

TL;DR

This paper provides the first theoretical convergence guarantees for Nesterov's accelerated gradient method in training deep linear neural networks, demonstrating its accelerated convergence over gradient descent.

Contribution

It offers the first theoretical analysis of NAG convergence in deep neural networks, including deep fully-connected and ResNet architectures, under over-parameterization.

Findings

NAG converges to the global minimum at a rate of (1 - O(1/√κ))^t

NAG achieves faster convergence than gradient descent (GD)

The analysis extends to deep linear ResNets, showing similar convergence results.

Abstract

Momentum methods, including heavy-ball~(HB) and Nesterov's accelerated gradient~(NAG), are widely used in training neural networks for their fast convergence. However, there is a lack of theoretical guarantees for their convergence and acceleration since the optimization landscape of the neural network is non-convex. Nowadays, some works make progress towards understanding the convergence of momentum methods in an over-parameterized regime, where the number of the parameters exceeds that of the training instances. Nonetheless, current results mainly focus on the two-layer neural network, which are far from explaining the remarkable success of the momentum methods in training deep neural networks. Motivated by this, we investigate the convergence of NAG with constant learning rate and momentum parameter in training two architectures of deep linear networks: deep fully-connected linear neural networks and deep linear ResNets. Based on the over-parameterization regime, we first analyze the residual dynamics induced by the training trajectory of NAG for a deep fully-connected linear neural network under the random Gaussian initialization. Our results show that NAG can converge to the global minimum at a $(1 - \mathcal{O}(1/\sqrt{\kappa}))^t$ rate, where $t$ is the iteration number and $\kappa > 1$ is a constant depending on the condition number of the feature matrix. Compared to the $(1 - \mathcal{O}(1/{\kappa}))^t$ rate of GD, NAG achieves an acceleration over GD. To the best of our knowledge, this is the first theoretical guarantee for the convergence of NAG to the global minimum in training deep neural networks. Furthermore, we extend our analysis to deep linear ResNets and derive a similar convergence result.