A Discrete Variational Derivation of Accelerated Methods in Optimization

TL;DR

This paper introduces a variational integrator framework for deriving accelerated optimization algorithms, generalizing classical methods like Polyak's heavy ball and Nesterov's method through geometric and symplectic principles.

Contribution

It presents a novel variational approach to derive accelerated optimization algorithms, connecting geometric integration with momentum-based methods.

Findings

Derived new optimization methods using variational integrators.

Generalized Polyak's heavy ball and Nesterov methods.

Experimental results demonstrate effectiveness of the proposed methods.

Abstract

Many of the new developments in machine learning are connected with gradient-based optimization methods. Recently, these methods have been studied using a variational perspective. This has opened up the possibility of introducing variational and symplectic methods using geometric integration. In particular, in this paper, we introduce variational integrators which allow us to derive different methods for optimization. Using both, Hamilton's and Lagrange-d'Alembert's principle, we derive two families of respective optimization methods in one-to-one correspondence that generalize Polyak's heavy ball and the well known Nesterov accelerated gradient method, the second of which mimics the behavior of the first reducing the oscillations of classical momentum methods. However, since the systems considered are explicitly time-dependent, the preservation of symplecticity of autonomous systems occurs here solely on the fibers. Several experiments exemplify the result.