Parameter Symmetry Potentially Unifies Deep Learning Theory

TL;DR

This paper advocates for the study of parameter symmetries as a unifying principle to understand hierarchical learning phenomena in large AI systems, linking dynamics, complexity, and representations.

Contribution

It proposes that parameter symmetry breaking and restoration could unify existing fragmented theories of neural network behavior.

Findings

Parameter symmetry may unify learning dynamics, complexity, and representation hierarchies.

Symmetry principles from physics could be fundamental in AI theory.

Position paper synthesizes prior theories to support this unifying hypothesis.

Abstract

The dynamics of learning in modern large AI systems is hierarchical, often characterized by abrupt, qualitative shifts akin to phase transitions observed in physical systems. While these phenomena hold promise for uncovering the mechanisms behind neural networks and language models, existing theories remain fragmented, addressing specific cases. In this position paper, we advocate for the crucial role of the research direction of parameter symmetries in unifying these fragmented theories. This position is founded on a centralizing hypothesis for this direction: parameter symmetry breaking and restoration are the unifying mechanisms underlying the hierarchical learning behavior of AI models. We synthesize prior observations and theories to argue that this direction of research could lead to a unified understanding of three distinct hierarchies in neural networks: learning dynamics, model complexity, and representation formation. By connecting these hierarchies, our position paper elevates symmetry -- a cornerstone of theoretical physics -- to become a potential fundamental principle in modern AI.