Spontaneous symmetry breaking and Goldstone modes for deep information propagation

TL;DR

This paper explores how continuous symmetry breaking in neural networks leads to Goldstone-like modes that facilitate stable, long-distance information propagation, improving trainability and long-term memory in deep and recurrent models.

Contribution

It introduces the concept of Goldstone modes in neural networks with continuous symmetry, demonstrating their role in stable information flow and long-term memory both analytically and empirically.

Findings

Goldstone-like modes enable coherent signal propagation across network depth.

Networks with these modes show improved trainability and representational diversity.

Recurrent networks benefit from these modes for enhanced long-term memory.

Abstract

In physical systems, whenever a continuous symmetry is spontaneously broken, the system possesses excitations called Goldstone modes, which allow coherent information propagation over long distances and times. In this work, we study deep neural networks whose internal layers are equivariant under a continuous symmetry and may therefore support analogous Goldstone-like degrees of freedom. We demonstrate, both analytically and empirically, that these degrees of freedom enable coherent signal propagation across depth and recurrent iterations, providing a mechanism for stable information flow without relying on architectural stabilizers such as residual connections or normalization. In feedforward networks, this results in improved trainability and representational diversity across layers. In recurrent settings, we demonstrate the same mechanism is valuable for long-term memory by propagating information over recurrent iterations, thereby improving performance of RNNs and GRUs on long-sequence modeling tasks.