Deep Linear Networks for Matrix Completion -- An Infinite Depth Limit

TL;DR

This paper explores the geometric properties of deep linear networks, especially in the infinite depth limit, and links their Riemannian geometry to implicit regularization in matrix completion tasks.

Contribution

It extends the geometric framework of DLNs to infinite depth, providing explicit volume form expressions and analyzing their role in implicit regularization for matrix completion.

Findings

Explicit volume form expressions for infinite depth DLNs

Connection between Riemannian geometry and training asymptotics

Implicit regularization bias towards high volume regions

Abstract

The deep linear network (DLN) is a model for implicit regularization in gradient based optimization of overparametrized learning architectures. Training the DLN corresponds to a Riemannian gradient flow, where the Riemannian metric is defined by the architecture of the network and the loss function is defined by the learning task. We extend this geometric framework, obtaining explicit expressions for the volume form, including the case when the network has infinite depth. We investigate the link between the Riemannian geometry and the training asymptotics for matrix completion with rigorous analysis and numerics. We propose that implicit regularization is a result of bias towards high state space volume.