Performance Estimation of second-order optimization methods on classes of univariate functions

TL;DR

This paper introduces a new framework for precisely analyzing the worst-case performance of second-order optimization methods on univariate functions, improving convergence bounds and establishing fundamental lower bounds.

Contribution

It develops a generic technique for deriving interpolation conditions for various univariate functions and uses it within the Performance Estimation framework to analyze second-order methods.

Findings

Improved convergence rates for second-order methods on univariate functions.

Established univariate lower bounds applicable to multivariate cases.

Provided tight performance guarantees for Newton's method variants.

Abstract

We develop a principled approach to obtain exact computer-aided worst-case guarantees on the performance of second-order optimization methods on classes of univariate functions. We first present a generic technique to derive interpolation conditions for a wide range of univariate functions, and use it to obtain such conditions for generalized self-concordant functions (including self-concordant and quasi-self-concordant functions) and functions with Lipschitz Hessian (both convex and non-convex). We then exploit these conditions within the Performance Estimation framework to tightly analyze the convergence of second-order methods on univariate functions, including (Cubic Regularized) Newton's method and several of its variants. Thereby, we improve on existing convergence rates, exhibit univariate lower bounds (that thus hold in the multivariate case), and analyze the performance of these methods with respect to the same criteria.