Implicit Bias and Loss of Plasticity in Matrix Completion: Depth Promotes Low-Rankness

TL;DR

This paper investigates how increasing depth in deep matrix factorization models enhances low-rank bias and coupling dynamics, explaining phenomena like plasticity loss and providing theoretical insights into convergence behaviors.

Contribution

It reveals that network depth promotes low-rank bias through coupled dynamics and resolves open questions about convergence in deep linear networks.

Findings

Depth ≥ 3 networks exhibit coupling unless initialized diagonally.

Coupled dynamics lead to convergence to rank-1 solutions.

Deep models avoid plasticity loss due to their low-rank bias.

Abstract

We study matrix completion via deep matrix factorization (a.k.a. deep linear neural networks) as a simplified testbed to examine how network depth influences training dynamics. Despite the simplicity and importance of the problem, prior theory largely focuses on shallow (depth-2) models and does not fully explain the implicit low-rank bias observed in deeper networks. We identify coupled dynamics as a key mechanism behind this bias and show that it intensifies with increasing depth. Focusing on gradient flow under block-diagonal observations, we prove: (a) networks of depth $\geq 3$ exhibit coupling unless initialized diagonally, and (b) convergence to rank-1 occurs if and only if the dynamics is coupled -- resolving an open question by Menon (2024) for a family of initializations. We also revisit the loss of plasticity phenomenon in matrix completion (Kleinman et al., 2024), where pre-training on few observations and resuming with more degrades performance. We show that deep models avoid plasticity loss due to their low-rank bias, whereas depth-2 networks pre-trained under decoupled dynamics fail to converge to low-rank, even when resumed training (with additional data) satisfies the coupling condition -- shedding light on the mechanism behind this phenomenon.