Learning deep linear neural networks: Riemannian gradient flows and convergence to global minimizers

TL;DR

This paper analyzes the convergence behavior of gradient flows in deep linear neural networks, showing they always reach critical points and often converge to global minima on matrix manifolds, using Riemannian geometry.

Contribution

It introduces a Riemannian geometric perspective to study gradient flows in deep linear networks and proves convergence to global minima for almost all initializations.

Findings

Gradient flows always converge to critical points.

Almost all initializations lead to convergence at global minima.

The analysis uses Riemannian geometry on matrix manifolds.

Abstract

We study the convergence of gradient flows related to learning deep linear neural networks (where the activation function is the identity map) from data. In this case, the composition of the network layers amounts to simply multiplying the weight matrices of all layers together, resulting in an overparameterized problem. The gradient flow with respect to these factors can be re-interpreted as a Riemannian gradient flow on the manifold of rank-$r$ matrices endowed with a suitable Riemannian metric. We show that the flow always converges to a critical point of the underlying functional. Moreover, we establish that, for almost all initializations, the flow converges to a global minimum on the manifold of rank $k$ matrices for some $k\leq r$.