Gradient Descent for Deep Matrix Factorization: Dynamics and Implicit Bias towards Low Rank

TL;DR

This paper analyzes the dynamics of gradient descent in deep linear networks, revealing how it implicitly biases solutions towards low-rank matrices, which helps explain generalization in over-parameterized models.

Contribution

It provides a rigorous analysis of gradient descent dynamics and characterizes the spectrum convergence, offering insights into implicit bias towards low-rank solutions.

Findings

Gradient descent dynamics favor low-rank solutions over time.

Effective rank of iterates can guide early stopping criteria.

Empirical evidence supports implicit bias in matrix sensing and random initialization.

Abstract

In deep learning, it is common to use more network parameters than training points. In such scenarioof over-parameterization, there are usually multiple networks that achieve zero training error so that thetraining algorithm induces an implicit bias on the computed solution. In practice, (stochastic) gradientdescent tends to prefer solutions which generalize well, which provides a possible explanation of thesuccess of deep learning. In this paper we analyze the dynamics of gradient descent in the simplifiedsetting of linear networks and of an estimation problem. Although we are not in an overparameterizedscenario, our analysis nevertheless provides insights into the phenomenon of implicit bias. In fact, wederive a rigorous analysis of the dynamics of vanilla gradient descent, and characterize the dynamicalconvergence of the spectrum. We are able to accurately locate time intervals where the effective rankof the iterates is close to the effective rank of a low-rank projection of the ground-truth matrix. Inpractice, those intervals can be used as criteria for early stopping if a certain regularity is desired. Wealso provide empirical evidence for implicit bias in more general scenarios, such as matrix sensing andrandom initialization. This suggests that deep learning prefers trajectories whose complexity (measuredin terms of effective rank) is monotonically increasing, which we believe is a fundamental concept for thetheoretical understanding of deep learning.