Essential Convergence Rates of Continuous-Time Models for Optimization Methods

TL;DR

This paper introduces the concept of essential convergence rates for continuous-time optimization models, establishing that discretized methods cannot surpass these rates, thereby clarifying the true speed limits of such methods.

Contribution

The paper defines the essential convergence rate to address rate ambiguity caused by time rescaling and proves that discretized methods cannot exceed this rate.

Findings

Introduces the notion of essential convergence rate.

Proves that discretized methods cannot outperform the essential rate.

Clarifies the fundamental speed limits of continuous-time optimization models.

Abstract

Designing and analyzing optimization methods via continuous-time models expressed as ordinary differential equations (ODEs) is a promising approach for its intuitiveness and simplicity. A key concern, however, is that the convergence rates of such models can be arbitrarily modified by time rescaling, rendering the task of seeking ODEs with ``fast'' convergence meaningless. To eliminate this ambiguity of the rates, we introduce the notion of the essential convergence rate. We justify this notion by proving that, under appropriate assumptions on discretization, no method obtained by discretizing an ODE can achieve a faster rate than its essential convergence rate.