Essential convergence rate of ordinary differential equations appearing in optimization

TL;DR

This paper introduces the concept of an 'essential convergence rate' for ODEs related to optimization, highlighting the balance between acceleration via time scaling and deceleration in discretization, which determines the attainable convergence rate.

Contribution

It establishes a fundamental limit on the convergence rates achievable through discretized ODEs in optimization, considering stability constraints.

Findings

Defines the 'essential convergence rate' as a fundamental limit.

Shows that acceleration by time scaling leads to deceleration in discretization.

Balances continuous and discrete perspectives to identify attainable convergence rates.

Abstract

Some continuous optimization methods can be connected to ordinary differential equations (ODEs) by taking continuous limits, and their convergence rates can be explained by the ODEs. However, since such ODEs can achieve any convergence rate by time scaling, the correspondence is not as straightforward as usually expected, and deriving new methods through ODEs is not quite direct. In this letter, we pay attention to stability restriction in discretizing ODEs and show that acceleration by time scaling basically implies deceleration in discretization; they balance out so that we can define an attainable unique convergence rate which we call an "essential convergence rate".