Symmetry & Critical Points

TL;DR

This paper investigates the nature of critical points in symmetric functions, revealing that symmetric critical points tend to be surrounded by symmetry-breaking points, which impacts optimization in neural networks.

Contribution

It introduces a mathematical mechanism showing that symmetric critical points are typically surrounded by symmetry-breaking points, affecting invariant nonconvex function minimization.

Findings

Symmetric critical points are often adjacent to symmetry-breaking points.

Implications for neural network optimization and invariant nonconvex functions.

Provides a theoretical foundation for understanding symmetry in critical points.

Abstract

Critical points of an invariant function may or may not be symmetric. We prove, however, that if a symmetric critical point exists, those adjacent to it are generically symmetry breaking. This mathematical mechanism is shown to carry important implications for our ability to efficiently minimize invariant nonconvex functions, in particular those associated with neural networks.