Implicit Bias in Matrix Factorization and its Explicit Realization in a New Architecture

TL;DR

This paper investigates the implicit low-rank bias in matrix factorization under gradient descent and introduces a new model with explicit constraints that reliably produces low-rank solutions, extending the concept to neural networks.

Contribution

The paper presents a novel factorization model with constrained factors and a diagonal component, explicitly capturing the implicit bias and extending it to neural network architectures.

Findings

The new model consistently yields truly low-rank solutions.

Experiments demonstrate the model's strong implicit bias and stability.

The neural network extension achieves competitive performance with low-rank representations.

Abstract

Gradient descent for matrix factorization exhibits an implicit bias toward approximately low-rank solutions. While existing theories often assume the boundedness of iterates, empirically the bias persists even with unbounded sequences. This reflects a dynamic where factors develop low-rank structure while their magnitudes increase, tending to align with certain directions. To capture this behavior in a stable way, we introduce a new factorization model: $X\approx UDV^\top$, where $U$ and $V$ are constrained within norm balls, while $D$ is a diagonal factor allowing the model to span the entire search space. Experiments show that this model consistently exhibits a strong implicit bias, yielding truly (rather than approximately) low-rank solutions. Extending the idea to neural networks, we introduce a new model featuring constrained layers and diagonal components that achieves competitive performance on various regression and classification tasks while producing lightweight, low-rank representations.