A Dynamics Theory of Implicit Regularization in Deep Low-Rank Matrix Factorization

TL;DR

This paper introduces a new theoretical approach to understanding how deep matrix factorization implicitly regularizes solutions, linking saddle point escaping stages to matrix rank and providing insights into the training dynamics of deep neural networks.

Contribution

It proposes a landscape analysis approach for discrete gradient dynamics, establishing a theoretical connection between saddle point escaping stages and matrix rank in deep matrix factorization.

Findings

DMF converges to second-order critical points after R saddle point escaping stages for rank-R matrices

Theoretical results are experimentally verified on low-rank matrix reconstruction

Provides new insights into implicit regularization in deep learning

Abstract

Implicit regularization is an important way to interpret neural networks. Recent theory starts to explain implicit regularization with the model of deep matrix factorization (DMF) and analyze the trajectory of discrete gradient dynamics in the optimization process. These discrete gradient dynamics are relatively small but not infinitesimal, thus fitting well with the practical implementation of neural networks. Currently, discrete gradient dynamics analysis has been successfully applied to shallow networks but encounters the difficulty of complex computation for deep networks. In this work, we introduce another discrete gradient dynamics approach to explain implicit regularization, i.e. landscape analysis. It mainly focuses on gradient regions, such as saddle points and local minima. We theoretically establish the connection between saddle point escaping (SPE) stages and the matrix rank in DMF. We prove that, for a rank-R matrix reconstruction, DMF will converge to a second-order critical point after R stages of SPE. This conclusion is further experimentally verified on a low-rank matrix reconstruction problem. This work provides a new theory to analyze implicit regularization in deep learning.