Ubiquitous Symmetry at Critical Points Across Diverse Optimization Landscapes

TL;DR

This paper investigates the pervasive presence of symmetry at critical points across various optimization landscapes, extending prior neural network findings to broader mathematical contexts and introducing a new symmetry measure.

Contribution

It generalizes the understanding of symmetry at critical points to diverse spaces and introduces a novel symmetry measure revealing deeper structures.

Findings

Critical points exhibit significant symmetry across different landscapes.

All observed critical points possess non-trivial symmetry.

A new symmetry measure uncovers additional symmetry structures.

Abstract

Symmetry plays a crucial role in understanding the properties of mathematical structures and optimization problems. Recent work has explored this phenomenon in the context of neural networks, where the loss function is invariant under column and row permutations of the network weights. It has been observed that local minima exhibit significant symmetry with respect to the network weights (invariance to row and column permutations). And moreover no critical point was found that lacked symmetry. We extend this line of inquiry by investigating symmetry phenomena in real-valued loss functions defined on a broader class of spaces. We will introduce four more cases: the projective case over a finite field, the octahedral graph case, the perfect matching case, and the particle attraction case. We show that as in the neural network case, all the critical points observed have non-trivial symmetry. Finally we introduce a new measure of symmetry in the system and show that it reveals additional symmetry structures not captured by the previous measure.