Gradient Descent Happens in a Tiny Subspace

TL;DR

This paper demonstrates that in large-scale deep learning, gradients rapidly concentrate in a small subspace spanned by top Hessian eigenvectors, which remains stable over training and may explain the dynamics of gradient descent.

Contribution

It reveals that gradient descent primarily occurs within a tiny, stable subspace defined by top Hessian eigenvectors, offering new insights into optimization in deep learning.

Findings

Gradients converge to a small subspace after short training

The subspace is spanned by top eigenvectors of the Hessian

Gradient descent mainly occurs within this subspace

Abstract

We show that in a variety of large-scale deep learning scenarios the gradient dynamically converges to a very small subspace after a short period of training. The subspace is spanned by a few top eigenvectors of the Hessian (equal to the number of classes in the dataset), and is mostly preserved over long periods of training. A simple argument then suggests that gradient descent may happen mostly in this subspace. We give an example of this effect in a solvable model of classification, and we comment on possible implications for optimization and learning.