Exponential Convergence Time of Gradient Descent for One-Dimensional Deep Linear Neural Networks

TL;DR

This paper proves that gradient descent on deep linear neural networks in one dimension takes exponentially longer to converge as the network depth increases, highlighting a significant challenge in understanding deep network training.

Contribution

It establishes that convergence time for gradient descent scales exponentially with network depth in one-dimensional deep linear neural networks under standard initializations.

Findings

Convergence time scales exponentially with depth in 1D networks.

Empirical evidence suggests similar behavior in higher dimensions.

Highlights potential difficulties in training deep linear networks.

Abstract

We study the dynamics of gradient descent on objective functions of the form $f(\prod_{i=1}^{k} w_i)$ (with respect to scalar parameters $w_1,\ldots,w_k$), which arise in the context of training depth-$k$ linear neural networks. We prove that for standard random initializations, and under mild assumptions on $f$, the number of iterations required for convergence scales exponentially with the depth $k$. We also show empirically that this phenomenon can occur in higher dimensions, where each $w_i$ is a matrix. This highlights a potential obstacle in understanding the convergence of gradient-based methods for deep linear neural networks, where $k$ is large.