Gradient-Normalized Smoothness for Optimization with Approximate Hessians

TL;DR

This paper introduces Gradient-Normalized Smoothness, a new concept that enables optimization algorithms with approximate Hessians to achieve fast, global convergence across various function classes, including non-convex problems.

Contribution

It proposes Gradient-Normalized Smoothness, a universal measure linking Hessian approximation to gradient behavior, leading to algorithms with guaranteed global convergence rates for diverse objectives.

Findings

Achieves state-of-the-art convergence rates for functions with Hölder-continuous Hessians.

Provides global linear rates for logistic regression and softmax with approximate Hessians.

Extends to non-convex optimization using Fisher and Gauss-Newton approximations.

Abstract

In this work, we develop new optimization algorithms that use approximate second-order information combined with the gradient regularization technique to achieve fast global convergence rates for both convex and non-convex objectives. The key innovation of our analysis is a novel notion called Gradient-Normalized Smoothness, which characterizes the maximum radius of a ball around the current point that yields a good relative approximation of the gradient field. Our theory establishes a natural intrinsic connection between Hessian approximation and the linearization of the gradient. Importantly, Gradient-Normalized Smoothness does not depend on the specific problem class of the objective functions, while effectively translating local information about the gradient field and Hessian approximation into the global behavior of the method. This new concept equips approximate second-order algorithms with universal global convergence guarantees, recovering state-of-the-art rates for functions with H\"older-continuous Hessians and third derivatives, quasi-self-concordant functions, as well as smooth classes in first-order optimization. These rates are achieved automatically and extend to broader classes, such as generalized self-concordant functions. We demonstrate direct applications of our results for global linear rates in logistic regression and softmax problems with approximate Hessians, as well as in non-convex optimization using Fisher and Gauss-Newton approximations.