Entropic Regularization in the Deep Linear Network

TL;DR

This paper analyzes the effects of entropic regularization on deep linear networks, characterizing equilibria, gradient flow dynamics, and the concavity of entropy in different geometric settings.

Contribution

It provides a detailed theoretical analysis of entropic regularization in DLNs, including explicit solutions for gradient flow and insights into the geometry of the entropy landscape.

Findings

Equilibria are minimizers forming an orthogonal orbit.

Gradient flow reduces to a one-dimensional ODE with explicit relaxation rates.

Entropy is strictly concave in Euclidean geometry but not in Riemannian geometry.

Abstract

We study regularization for the deep linear network (DLN) using the entropy formula introduced in arXiv:2509.09088. The equilibria and gradient flow of the free energy on the Riemannian manifold of end-to-end maps of the DLN are characterized for energies that depend symmetrically on the singular values of the end-to-end matrix. The only equilibria are minimizers and the set of minimizers is an orbit of the orthogonal group. In contrast with random matrix theory there is no singular value repulsion. The corresponding gradient flow reduces to a one-dimensional ordinary differential equation whose solution gives explicit relaxation rates toward the minimizers. We also study the concavity of the entropy in the chamber of singular values. The entropy is shown to be strictly concave in the Euclidean geometry on the chamber but not in the Riemannian geometry defined by the DLN metric.