Newton Method Revisited: Global Convergence Rates up to $\mathcal {O}\left(k^{-3} \right)$ for Stepsize Schedules and Linesearch Procedures

TL;DR

This paper advances the understanding of Newton methods by establishing global convergence rates up to (k^{-3}) for various stepsize strategies and linesearch procedures, applicable to convex functions with Hölder continuous derivatives.

Contribution

It introduces new stepsize schedules and linesearch techniques that achieve fast global convergence guarantees, even with unknown smoothness parameters.

Findings

Achieves (k^{-3}) convergence rate with simple stepsize schedules.

Develops linesearch and backtracking methods with provable convergence under unknown smoothness.

Provides strong convergence guarantees for Newton method with exact linesearch.

Abstract

This paper investigates the global convergence of stepsized Newton methods for convex functions with H\"older continuous Hessians or third derivatives. We propose several simple stepsize schedules with fast global convergence guarantees, up to $\mathcal {O}\left(k^{-3} \right)$. For cases with multiple plausible smoothness parameterizations or an unknown smoothness constant, we introduce a stepsize linesearch and a backtracking procedure with provable convergence as if the optimal smoothness parameters were known in advance. Additionally, we present strong convergence guarantees for the practically popular Newton method with exact linesearch.