Noether: The More Things Change, the More Stay the Same
Grzegorz G{\l}uch, R\"udiger Urbanke

TL;DR
This paper systematically explores how symmetries in neural networks influence training dynamics, revealing conserved quantities and restrictions on gradient paths, thereby deepening understanding of neural network behavior and over-parametrization effects.
Contribution
It provides a formal analysis of symmetry in neural networks using Noether's theorem, linking symmetries to conserved quantities and restrictions on training paths.
Findings
Symmetries lead to boundedness of weights and balance equations.
Data augmentation induces momentum-like restrictions.
Time symmetry relates to the Neural Tangent Kernel.
Abstract
Symmetries have proven to be important ingredients in the analysis of neural networks. So far their use has mostly been implicit or seemingly coincidental. We undertake a systematic study of the role that symmetry plays. In particular, we clarify how symmetry interacts with the learning algorithm. The key ingredient in our study is played by Noether's celebrated theorem which, informally speaking, states that symmetry leads to conserved quantities (e.g., conservation of energy or conservation of momentum). In the realm of neural networks under gradient descent, model symmetries imply restrictions on the gradient path. E.g., we show that symmetry of activation functions leads to boundedness of weight matrices, for the specific case of linear activations it leads to balance equations of consecutive layers, data augmentation leads to gradient paths that have "momentum"-type restrictions,…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsStochastic Gradient Optimization Techniques · Neural Networks and Applications · Model Reduction and Neural Networks
