Riemannian Lyapunov Optimizer: A Unified Framework for Optimization

TL;DR

This paper introduces Riemannian Lyapunov Optimizers, a unified geometric framework derived from control theory that systematically designs stable and effective optimization algorithms for machine learning.

Contribution

It presents a novel control-theoretic approach to optimizer design, unifying classic algorithms within a Riemannian geometric framework and enabling principled creation of new optimizers.

Findings

State-of-the-art performance on large-scale benchmarks

Systematic derivation of optimizers from control theory

Validation through geometric diagnostics

Abstract

We introduce Riemannian Lyapunov Optimizers (RLOs), a family of optimization algorithms that unifies classic optimizers within one geometric framework. Unlike heuristic improvements to existing optimizers, RLOs are systematically derived from a novel control-theoretic framework that reinterprets optimization as an extended state discrete-time controlled dynamical system on a Riemannian parameter manifold. Central to this framework is the identification of a Normally Attracting Invariant Manifold (NAIM), which organizes training dynamics into two distinct stages: rapid alignment of the speed state to a target graph, followed by controlled evolution within it. We formalize this by constructing a strict Lyapunov function that certifies convergence to a target manifold. This perspective yields a constructive ``optimizer generator" that not only recovers classic algorithms but enables the principled design of RLOs. We validate our theory via geometric diagnostics and demonstrate that grounding optimizer design in control theory yields state-of-the-art performance in large-scale benchmarks. Overall, RLOs bridge control theory and modern machine learning optimization, providing a unified language and a systematic toolkit for designing stable, effective optimizers.