Global Convergence Analysis of Deep Linear Networks with A One-neuron Layer

TL;DR

This paper provides a comprehensive non-local convergence analysis of deep linear networks with a single-neuron layer, extending previous work to arbitrary initializations and detailing convergence paths and rates.

Contribution

It introduces a novel non-local convergence framework for deep linear networks with one-neuron layers, surpassing prior lazy training assumptions.

Findings

Characterizes convergence points including saddle points and global minima.

Establishes convergence rates to the global minimizer in stages.

Defines the rank-stable set and global minimizer set for the first time in this context.

Abstract

In this paper, we follow Eftekhari's work to give a non-local convergence analysis of deep linear networks. Specifically, we consider optimizing deep linear networks which have a layer with one neuron under quadratic loss. We describe the convergent point of trajectories with arbitrary starting point under gradient flow, including the paths which converge to one of the saddle points or the original point. We also show specific convergence rates of trajectories that converge to the global minimizer by stages. To achieve these results, this paper mainly extends the machinery in Eftekhari's work to provably identify the rank-stable set and the global minimizer convergent set. We also give specific examples to show the necessity of our definitions. Crucially, as far as we know, our results appear to be the first to give a non-local global analysis of linear neural networks from arbitrary initialized points, rather than the lazy training regime which has dominated the literature of neural networks, and restricted benign initialization in Eftekhari's work. We also note that extending our results to general linear networks without one hidden neuron assumption remains a challenging open problem.