When is the average number of saddle points typical?

TL;DR

This paper investigates the difference between typical and average counts of stationary points in complex functions, showing that equilibrium heuristics do not always predict the behavior of saddles and minima in correlated Gaussian models.

Contribution

It demonstrates that quenched and annealed averages can diverge for saddles and minima, challenging assumptions based on equilibrium heuristics in correlated function models.

Findings

Quenched and annealed averages differ for certain saddles and minima.

Equilibrium heuristics do not reliably predict non-equilibrium behavior.

Conditions for correlations between saddles are identified.

Abstract

A common measure of a function's complexity is the count of its stationary points. For complicated functions, this count grows exponentially with the volume and dimension of their domain. In practice, the count is averaged over a class of functions (the annealed average), but the large numbers involved can produce averages biased by extremely rare samples. Typical counts are reliably found by taking the average of the logarithm (the quenched average), which is more difficult and not often done in practice. When most stationary points are uncorrelated with each other, quenched and anneals averages are equal. Equilibrium heuristics can guarantee when most of the lowest minima will be uncorrelated. We show that these equilibrium heuristics cannot be used to draw conclusions about other minima and saddles by producing examples among Gaussian-correlated functions on the hypersphere where the count of certain saddles and minima has different quenched and annealed averages, despite being guaranteed `safe' in the equilibrium setting. We determine conditions for the emergence of nontrivial correlations between saddles, and discuss the implications for the geometry of those functions and what out-of-equilibrium settings might be affected.