Towards faster first order methods: A continuous-time model to interpolate between speed and function value restart

TL;DR

This paper proposes a novel restart scheme for continuous inertial dynamics that improves convergence rates of first-order methods by interpolating between speed and function value restarts without requiring strong convexity knowledge.

Contribution

It introduces a generalized restart routine that delays restarting, preserves convergence, and enhances accelerated first-order algorithms through a continuous-time model.

Findings

Linear convergence rate for function values along restarted trajectories

Improved convergence rates in numerical experiments for dynamical systems

Effective interpolation between speed and function value restarts

Abstract

We introduce a new restarting scheme for a continuous inertial dynamics with Hessian driven-damping, and establish a linear convergence rate for the function values along the restarted trajectories. The proposed routine is implemented without knowing the strong convexity parameter, and is a generalization of existing speed restart schemes. It interpolates between speed and function value restarts, considerably delaying the restarting time, while preserving convergence and function value decrease. Numerical experiments show an improvement in the convergence rates for both continuous-time dynamical systems, and the associated accelerated first-order algorithms derived via time discretization.