On Symplectic Optimization

TL;DR

This paper introduces a systematic framework for converting continuous-time Hamiltonian dynamics into discrete optimization algorithms using symplectic integration, enhancing the theoretical understanding and practical design of accelerated gradient methods.

Contribution

It provides a novel methodology that leverages symplectic integration and Hamiltonian systems to develop accelerated optimization algorithms with preserved oracle rates.

Findings

Framework successfully converts continuous dynamics to discrete algorithms.

Retains oracle convergence rates in the discretization process.

Bridges the gap between Hamiltonian systems and optimization algorithms.

Abstract

Accelerated gradient methods have had significant impact in machine learning -- in particular the theoretical side of machine learning -- due to their ability to achieve oracle lower bounds. But their heuristic construction has hindered their full integration into the practical machine-learning algorithmic toolbox, and has limited their scope. In this paper we build on recent work which casts acceleration as a phenomenon best explained in continuous time, and we augment that picture by providing a systematic methodology for converting continuous-time dynamics into discrete-time algorithms while retaining oracle rates. Our framework is based on ideas from Hamiltonian dynamical systems and symplectic integration. These ideas have had major impact in many areas in applied mathematics, but have not yet been seen to have a relationship with optimization.