Continuous Newton-like Methods featuring Inertia and Variable Mass

TL;DR

This paper introduces a novel dynamical system with a variable mass parameter that interpolates between inertial and Newton's methods, providing new insights into accelerated optimization and convergence guarantees.

Contribution

It proposes a new second-order dynamical system with a time-dependent mass parameter, connecting inertial Newton-like methods with regularized Newton methods, and analyzes its convergence properties.

Findings

Variable mass influences asymptotic convergence rate.

The system can be tuned for acceleration in optimization.

Numerical experiments validate theoretical results.

Abstract

We introduce a new dynamical system, at the interface between second-order dynamics with inertia and Newton's method. This system extends the class of inertial Newton-like dynamics by featuring a time-dependent parameter in front of the acceleration, called variable mass. For strongly convex optimization, we provide guarantees on how the Newtonian and inertial behaviors of the system can be non-asymptotically controlled by means of this variable mass. A connection with the Levenberg--Marquardt (or regularized Newton's) method is also made. We then show the effect of the variable mass on the asymptotic rate of convergence of the dynamics, and in particular, how it can turn the latter into an accelerated Newton method. We provide numerical experiments supporting our findings. This work represents a significant step towards designing new algorithms that benefit from the best of both first- and second-order optimization methods.