An Equivariance Toolbox for Learning Dynamics

TL;DR

This paper introduces a comprehensive equivariance toolbox that extends classical symmetry analyses to second-order learning dynamics, providing new insights into the geometry of neural network loss landscapes and their relation to transformations.

Contribution

It develops a general framework for analyzing both first- and second-order constraints in neural learning dynamics, extending Noether-type analyses to broader transformations and discrete cases.

Findings

Unifies conservation laws and implicit biases under a single identity.

Predicts curvature properties and flat/sharp directions in loss landscapes.

Connects transformation structures to empirical optimization observations.

Abstract

Many theoretical results in deep learning can be traced to symmetry or equivariance of neural networks under parameter transformations. However, existing analyses are typically problem-specific and focus on first-order consequences such as conservation laws, while the implications for second-order structure remain less understood. We develop a general equivariance toolbox that yields coupled first- and second-order constraints on learning dynamics. The framework extends classical Noether-type analyses in three directions: from gradient constraints to Hessian constraints, from symmetry to general equivariance, and from continuous to discrete transformations. At the first order, our framework unifies conservation laws and implicit-bias relations as special cases of a single identity. At the second order, it provides structural predictions about curvature: which directions are flat or sharp, how the gradient aligns with Hessian eigenspaces, and how the loss landscape geometry reflects the underlying transformation structure. We illustrate the framework through several applications, recovering known results while also deriving new characterizations that connect transformation structure to modern empirical observations about optimization geometry.