Symmetry in Neural Network Parameter Spaces

TL;DR

This paper explores the role of symmetries in neural network parameter spaces, highlighting their impact on the loss landscape, learning dynamics, and theoretical understanding of deep learning models.

Contribution

It provides a comprehensive survey of existing research on parameter space symmetries, connecting them to learning theory and identifying future research directions.

Findings

Symmetries explain overparameterization redundancy.

Symmetries influence the shape of the loss landscape.

Connections between symmetry and generalization are emerging.

Abstract

Modern deep learning models are highly overparameterized, resulting in large sets of parameter configurations that yield the same outputs. A significant portion of this redundancy is explained by symmetries in the parameter space--transformations that leave the network function unchanged. These symmetries shape the loss landscape and constrain learning dynamics, offering a new lens for understanding optimization, generalization, and model complexity that complements existing theory of deep learning. This survey provides an overview of parameter space symmetry. We summarize existing literature, uncover connections between symmetry and learning theory, and identify gaps and opportunities in this emerging field.