SGD and Weight Decay Secretly Minimize the Rank of Your Neural Network

TL;DR

This paper reveals that SGD with weight decay inherently biases neural network weights towards low-rank solutions, especially with smaller batch sizes and higher learning rates, and explores its implications for generalization.

Contribution

It provides a theoretical and empirical analysis showing that SGD and weight decay induce a rank minimization bias without relying on data assumptions.

Findings

SGD with weight decay promotes low-rank weight matrices.

Smaller batch sizes and higher learning rates intensify the bias.

The bias has a marginal impact on test performance.

Abstract

We investigate the inherent bias of Stochastic Gradient Descent (SGD) toward learning low-rank weight matrices during the training of deep neural networks. Our results demonstrate that training with mini-batch SGD and weight decay induces a bias toward rank minimization in the weight matrices. Specifically, we show both theoretically and empirically that this bias becomes more pronounced with smaller batch sizes, higher learning rates, or stronger weight decay. Additionally, we predict and empirically confirm that weight decay is essential for this bias to occur. Unlike previous literature, our analysis does not rely on assumptions about the data, convergence, or optimality of the weight matrices, making it applicable to a wide range of neural network architectures of any width or depth. Finally, we empirically explore the connection between this bias and generalization, finding that it has a marginal effect on the test performance.