The geometry of the deep linear network

TL;DR

This paper explores the training dynamics of deep linear networks through geometric and thermodynamic frameworks, unifying existing results and linking to broader mathematical areas.

Contribution

It provides a unified geometric and thermodynamic analysis of deep linear networks, including invariant manifolds, Riemannian geometry, and stochastic gradient descent formulations.

Findings

Characterization of invariant manifolds in DLNs

Formulas for Boltzmann entropy and free energy

Connections between DLNs and other mathematical fields

Abstract

This article provides an expository account of training dynamics in the Deep Linear Network (DLN) from the perspective of the geometric theory of dynamical systems. Rigorous results by several authors are unified into a thermodynamic framework for deep learning. The analysis begins with a characterization of the invariant manifolds and Riemannian geometry in the DLN. This is followed by exact formulas for a Boltzmann entropy, as well as stochastic gradient descent of free energy using a Riemannian Langevin Equation. Several links between the DLN and other areas of mathematics are discussed, along with some open questions.