Optimization Insights into Deep Diagonal Linear Networks

TL;DR

This paper analyzes how gradient flow behaves in deep diagonal linear networks, revealing explicit convergence guarantees and the influence of parametrization and initialization on training efficiency.

Contribution

It provides a tractable analysis of deep diagonal linear networks, showing how their structure leads to well-behaved optimization dynamics and explicit convergence guarantees.

Findings

Gradient flow induces mirror-flow dynamics in effective parameters.

Exponential decay of loss under Polyak-Lojasiewicz condition.

Initialization and scaling influence training speed.

Abstract

Gradient-based methods successfully train highly overparameterized models in practice, even though the associated optimization problems are markedly nonconvex. Understanding the mechanisms that make such methods effective has become a central problem in modern optimization. To investigate this question in a tractable setting, we study Deep Diagonal Linear Networks. These are multilayer architectures with a reparameterization that preserves convexity in the effective parameter, while inducing a nontrivial geometry in the optimization landscape. Under mild initialization conditions, we show that gradient flow on the layer parameters induces a mirror-flow dynamic in the effective parameter space. This structural insight yields explicit convergence guarantees, including exponential decay of the loss under a Polyak-Lojasiewicz condition, and clarifies how the parametrization and initialization scale govern the training speed. Overall, our results demonstrate that deep diagonal over parameterizations, despite their apparent complexity, can endow standard gradient methods with well-behaved and interpretable optimization dynamics.