Width Provably Matters in Optimization for Deep Linear Neural Networks

TL;DR

This paper proves that sufficient width in deep linear neural networks guarantees convergence of gradient descent to a global minimum, highlighting the importance of width for effective optimization.

Contribution

It establishes a width threshold ensuring linear convergence of gradient descent in deep linear networks, linking width to data rank, condition number, and output dimension.

Findings

Gradient descent converges linearly with sufficient width.

Wide layers are necessary for efficient deep network optimization.

Narrow networks have exponential lower bounds on optimization complexity.

Abstract

We prove that for an $L$-layer fully-connected linear neural network, if the width of every hidden layer is $\tilde\Omega (L \cdot r \cdot d_{\mathrm{out}} \cdot \kappa^3 )$, where $r$ and $\kappa$ are the rank and the condition number of the input data, and $d_{\mathrm{out}}$ is the output dimension, then gradient descent with Gaussian random initialization converges to a global minimum at a linear rate. The number of iterations to find an $\epsilon$-suboptimal solution is $O(\kappa \log(\frac{1}{\epsilon}))$. Our polynomial upper bound on the total running time for wide deep linear networks and the $\exp\left(\Omega\left(L\right)\right)$ lower bound for narrow deep linear neural networks [Shamir, 2018] together demonstrate that wide layers are necessary for optimizing deep models.