Moduli space of optimization algorithms
Dmitry Pasechnyuk-Vilensky, Martin Tak\'a\v{c}

TL;DR
This paper develops a geometric framework for optimization algorithms, modeling them as discrete connections with curvature measures that relate to their stability, accuracy, and acceleration properties.
Contribution
It introduces a novel operator-theoretic formalism that unifies classical and advanced optimization methods through geometric and curvature concepts.
Findings
Classical algorithms are recovered as flat connection cases.
Higher-order variants with guaranteed stability are constructed.
Exact bounds and acceleration effects are characterized via geometric invariants.
Abstract
We introduce a geometric and operator-theoretic formalism viewing optimization algorithms as discrete connections on a space of update operators. Each iterative method is encoded by two coupled channels-drift and diffusion-whose algebraic curvature measures the deviation from ideal reversibility and determines the attainable order of accuracy. Flat connections correspond to methods whose updates commute up to higher order and thus achieve minimal numerical dissipation while preserving stability. The formalism recovers classical gradient, proximal, and momentum schemes as first-order flat cases and extends naturally to stochastic, preconditioned, and adaptive algorithms through perturbations controlled by curvature order. Explicit gauge corrections yield higher-order variants with guaranteed energy monotonicity and noise stability. The associated determinantal and isomonodromic…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsNumerical methods for differential equations · Stochastic Gradient Optimization Techniques · Advanced Optimization Algorithms Research
