Revisiting the Role of Euler Numerical Integration on Acceleration and Stability in Convex Optimization

TL;DR

This paper investigates the impact of Euler numerical integration methods on acceleration and stability in convex optimization, revealing that properties like symplecticity do not necessarily lead to acceleration.

Contribution

It introduces a new ODE framework showing that both explicit and semi-implicit Euler methods can produce accelerated algorithms, challenging existing beliefs about integrator quality and acceleration.

Findings

Semi-implicit Euler methods can lead to acceleration in convex optimization.

Properties like symplecticity do not guarantee acceleration.

Explicit Euler discretization also achieves acceleration.

Abstract

Viewing optimization methods as numerical integrators for ordinary differential equations (ODEs) provides a thought-provoking modern framework for studying accelerated first-order optimizers. In this literature, acceleration is often supposed to be linked to the quality of the integrator (accuracy, energy preservation, symplecticity). In this work, we propose a novel ordinary differential equation that questions this connection: both the explicit and the semi-implicit (a.k.a symplectic) Euler discretizations on this ODE lead to an accelerated algorithm for convex programming. Although semi-implicit methods are well-known in numerical analysis to enjoy many desirable features for the integration of physical systems, our findings show that these properties do not necessarily relate to acceleration.