The Law of Parsimony in Gradient Descent for Learning Deep Linear Networks

TL;DR

This paper uncovers a 'law of parsimony' in gradient descent dynamics for deep linear networks, showing that learning primarily occurs within a small invariant subspace, which enhances understanding and efficiency of training.

Contribution

The paper reveals a surprising invariant subspace property in deep linear networks' training dynamics, leading to more efficient training and insights into deep representation learning.

Findings

Gradient descent affects only a small subspace of singular vectors.

Training dynamics are confined to a low-dimensional invariant subspace.

Constructing smaller networks can replicate wider networks' benefits.

Abstract

Over the past few years, an extensively studied phenomenon in training deep networks is the implicit bias of gradient descent towards parsimonious solutions. In this work, we investigate this phenomenon by narrowing our focus to deep linear networks. Through our analysis, we reveal a surprising "law of parsimony" in the learning dynamics when the data possesses low-dimensional structures. Specifically, we show that the evolution of gradient descent starting from orthogonal initialization only affects a minimal portion of singular vector spaces across all weight matrices. In other words, the learning process happens only within a small invariant subspace of each weight matrix, despite the fact that all weight parameters are updated throughout training. This simplicity in learning dynamics could have significant implications for both efficient training and a better understanding of deep networks. First, the analysis enables us to considerably improve training efficiency by taking advantage of the low-dimensional structure in learning dynamics. We can construct smaller, equivalent deep linear networks without sacrificing the benefits associated with the wider counterparts. Second, it allows us to better understand deep representation learning by elucidating the linear progressive separation and concentration of representations from shallow to deep layers. We also conduct numerical experiments to support our theoretical results. The code for our experiments can be found at https://github.com/cjyaras/lawofparsimony.