Moduli space of optimization algorithms

TL;DR

This paper develops a geometric framework for optimization algorithms, modeling them as discrete connections with curvature measures that relate to their stability, accuracy, and acceleration properties.

Contribution

It introduces a novel operator-theoretic formalism that unifies classical and advanced optimization methods through geometric and curvature concepts.

Findings

Classical algorithms are recovered as flat connection cases.

Higher-order variants with guaranteed stability are constructed.

Exact bounds and acceleration effects are characterized via geometric invariants.

Abstract

We introduce a geometric and operator-theoretic formalism viewing optimization algorithms as discrete connections on a space of update operators. Each iterative method is encoded by two coupled channels-drift and diffusion-whose algebraic curvature measures the deviation from ideal reversibility and determines the attainable order of accuracy. Flat connections correspond to methods whose updates commute up to higher order and thus achieve minimal numerical dissipation while preserving stability. The formalism recovers classical gradient, proximal, and momentum schemes as first-order flat cases and extends naturally to stochastic, preconditioned, and adaptive algorithms through perturbations controlled by curvature order. Explicit gauge corrections yield higher-order variants with guaranteed energy monotonicity and noise stability. The associated determinantal and isomonodromic formulations yield exact nonasymptotic bounds and describe acceleration effects via Painlev\'e-type invariants and Stokes corrections.