Implicit Regularization Towards Rank Minimization in ReLU Networks

TL;DR

This paper investigates the relationship between implicit regularization in ReLU neural networks and rank minimization, revealing both limitations and conditions under which low-rank solutions are favored.

Contribution

It provides new theoretical and empirical insights into when gradient flow in ReLU networks promotes low-rank solutions, extending understanding beyond linear models.

Findings

Gradient flow may not minimize rank in ReLU networks for most datasets.

Deep ReLU networks are biased towards low-rank solutions under certain conditions.

Empirical evidence supports the theoretical results.

Abstract

We study the conjectured relationship between the implicit regularization in neural networks, trained with gradient-based methods, and rank minimization of their weight matrices. Previously, it was proved that for linear networks (of depth 2 and vector-valued outputs), gradient flow (GF) w.r.t. the square loss acts as a rank minimization heuristic. However, understanding to what extent this generalizes to nonlinear networks is an open problem. In this paper, we focus on nonlinear ReLU networks, providing several new positive and negative results. On the negative side, we prove (and demonstrate empirically) that, unlike the linear case, GF on ReLU networks may no longer tend to minimize ranks, in a rather strong sense (even approximately, for "most" datasets of size 2). On the positive side, we reveal that ReLU networks of sufficient depth are provably biased towards low-rank solutions in several reasonable settings.