Symmetry Induces Structure and Constraint of Learning

TL;DR

This paper reveals how symmetries in the loss function of neural networks influence learning behavior, leading to specific parameter constraints and explaining phenomena like sparsity and low rankness.

Contribution

It establishes a theoretical link between loss function symmetries and parameter constraints, providing insights into neural network behaviors and potential algorithmic applications.

Findings

Mirror-reflection symmetry induces parameter constraints.

Rescaling symmetry leads to sparsity in models.

Rotation symmetry results in low-rankness.

Abstract

Due to common architecture designs, symmetries exist extensively in contemporary neural networks. In this work, we unveil the importance of the loss function symmetries in affecting, if not deciding, the learning behavior of machine learning models. We prove that every mirror-reflection symmetry, with reflection surface $O$, in the loss function leads to the emergence of a constraint on the model parameters $\theta$: $O^T\theta =0$. This constrained solution becomes satisfied when either the weight decay or gradient noise is large. Common instances of mirror symmetries in deep learning include rescaling, rotation, and permutation symmetry. As direct corollaries, we show that rescaling symmetry leads to sparsity, rotation symmetry leads to low rankness, and permutation symmetry leads to homogeneous ensembling. Then, we show that the theoretical framework can explain intriguing phenomena, such as the loss of plasticity and various collapse phenomena in neural networks, and suggest how symmetries can be used to design an elegant algorithm to enforce hard constraints in a differentiable way.