Accelerated optimization algorithms and ordinary differential equations: the convex non Euclidean case

TL;DR

This paper introduces a novel accelerated optimization algorithm in a non-Euclidean setting, generalizing Nesterov's method and connecting it to differential equations and numerical discretizations.

Contribution

It proposes a new accelerated convex optimization algorithm in non-Euclidean spaces, extending existing methods and linking them to differential equations and Runge-Kutta schemes.

Findings

The algorithm achieves accelerated convergence rates.

Discretizations of the ODE outperform previous methods.

The approach unifies various equations and algorithms in the literature.

Abstract

We study the connections between ordinary differential equations and optimization algorithms in a non-Euclidean setting. We propose a novel accelerated algorithm for minimising convex functions over a convex constrained set. This algorithm is a natural generalization of Nesterov's accelerated gradient descent method to the non-Euclidean setting and can be interpreted as an additive Runge-Kutta algorithm. The algorithm can also be derived as a numerical discretization of the ODE appearing in Krichene et al. (2015a). We use Lyapunov functions to establish convergence rates for the ODE and show that the discretizations considered achieve acceleration beyond the setting studied in Krichene et al. (2015a). Finally, we discuss how the proposed algorithm connects to various equations and algorithms in the literature.