The Role of Symmetry in Optimizing Overparameterized Networks

TL;DR

This paper investigates how overparameterization and symmetry in neural networks influence optimization, revealing geometric effects that lead to better minima and faster convergence.

Contribution

It provides a theoretical framework linking symmetries, loss landscape geometry, and optimization benefits in overparameterized neural networks.

Findings

Symmetries act as diagonal preconditioning on the Hessian.

Overparameterization increases the probability of reaching global minima.

Wider networks exhibit lower top eigenvalues and faster convergence.

Abstract

Overparameterization is central to the success of deep learning, yet the mechanisms by which it improves optimization remain incompletely understood. We analyze weight-space symmetries in neural networks and show that overparameterization introduces additional symmetries that benefit optimization in two distinct ways. First, we prove that these symmetries act as a form of diagonal preconditioning on the Hessian, enabling the existence of better-conditioned minima within each equivalence class of functionally identical solutions. Second, we show that overparameterization increases the probability mass of global minima near typical initializations, making these favourable solutions more reachable. These results offer a potential link between loss landscape geometry and simplicity bias. Empirically, we observe wider networks have lower top eigenvalues, smaller condition numbers and faster convergence, matching our analysis. Our analysis provides a unified framework for understanding overparameterization and width growth as a geometric transformation of the loss landscape.