On the stability of optimization algorithms given by discretizations of the Euler-Lagrange ODE

TL;DR

This paper analyzes the stability of discretized Euler-Lagrange ODEs used in optimization algorithms, revealing conditions under which naive explicit-implicit discretizations become unstable and proposing stability bounds for different discretization schemes.

Contribution

It provides a theoretical stability analysis of discretizations of Euler-Lagrange equations in optimization, highlighting conditions for stability and instability in accelerated methods.

Findings

Naive explicit-implicit Euler discretization can be unstable for certain step sizes.

Stability bounds depend on the objective's smoothness, convexity, and discretization parameters.

Explicit and implicit Euler methods exhibit different stability behaviors under the studied conditions.

Abstract

The derivation of second-order ordinary differential equations (ODEs) as continuous-time limits of optimization algorithms has been shown to be an effective tool for the analysis of these algorithms. Additionally, discretizing generalizations of these ODEs can lead to new families of optimization methods. We study discretizations of an Euler-Lagrange equation which generate a large class of accelerated methods whose convergence rate is $O(\frac{1}{t^p})$ in continuous-time, where parameter $p$ is the order of the optimization method. Specifically, we address the question asking why a naive explicit-implicit Euler discretization of this solution produces an unstable algorithm, even for a strongly convex objective function. We prove that for a strongly convex $L$-smooth quadratic objective function and step size $\delta<\frac{1}{L}$, the naive discretization will exhibit stable behavior when the number of iterations $k$ satisfies the inequality $k < (\frac{4}{Lp^2 \delta^p})^{\frac{1}{p-2}}$. Additionally, we extend our analysis to the implicit and explicit Euler discretization methods to determine end behavior.