Symmetry Breaking in Neural Network Optimization: Insights from Input Dimension Expansion

TL;DR

This paper explores how symmetry breaking, induced by input dimension expansion, plays a crucial role in neural network optimization, offering new insights and metrics to improve network performance and design.

Contribution

It introduces the symmetry breaking hypothesis, demonstrates the benefits of input expansion, and develops a metric to quantify symmetry breaking in neural networks.

Findings

Input expansion improves neural network performance.

Symmetry breaking underpins techniques like dropout and batch normalization.

A new metric quantifies symmetry breaking to guide network design.

Abstract

Understanding the mechanisms behind neural network optimization is crucial for improving network design and performance. While various optimization techniques have been developed, a comprehensive understanding of the underlying principles that govern these techniques remains elusive. Specifically, the role of symmetry breaking, a fundamental concept in physics, has not been fully explored in neural network optimization. This gap in knowledge limits our ability to design networks that are both efficient and effective. Here, we propose the symmetry breaking hypothesis to elucidate the significance of symmetry breaking in enhancing neural network optimization. We demonstrate that a simple input expansion can significantly improve network performance across various tasks, and we show that this improvement can be attributed to the underlying symmetry breaking mechanism. We further develop a metric to quantify the degree of symmetry breaking in neural networks, providing a practical approach to evaluate and guide network design. Our findings confirm that symmetry breaking is a fundamental principle that underpins various optimization techniques, including dropout, batch normalization, and equivariance. By quantifying the degree of symmetry breaking, our work offers a practical technique for performance enhancement and a metric to guide network design without the need for complete datasets and extensive training processes.