Local Curvature Descent: Squeezing More Curvature out of Standard and Polyak Gradient Descent

TL;DR

This paper introduces three new local curvature descent methods that adaptively use curvature information to improve gradient descent, achieving better empirical performance without expensive second-order computations.

Contribution

The paper develops three novel local curvature descent algorithms (LCD1, LCD2, LCD3) that incorporate local curvature information into gradient descent, with theoretical guarantees and empirical improvements.

Findings

LCD methods outperform classical gradient descent in experiments

Theoretical analysis recovers known rates when curvature info is absent

LCD3 provides a closed-form iterative expression

Abstract

We contribute to the growing body of knowledge on more powerful and adaptive stepsizes for convex optimization, empowered by local curvature information. We do not go the route of fully-fledged second-order methods which require the expensive computation of the Hessian. Instead, our key observation is that, for some problems (e.g., when minimizing the sum of squares of absolutely convex functions), certain local curvature information is readily available, and can be used to obtain surprisingly powerful matrix-valued stepsizes, and meaningful theory. In particular, we develop three new methods$\unicode{x2013}$LCD1, LCD2 and LCD3$\unicode{x2013}$where the abbreviation stands for local curvature descent. While LCD1 generalizes gradient descent with fixed stepsize, LCD2 generalizes gradient descent with Polyak stepsize. Our methods enhance these classical gradient descent baselines with local curvature information, and our theory recovers the known rates in the special case when no curvature information is used. Our last method, LCD3, is a variable metric version of LCD2; this feature leads to a closed-form expression for the iterates. Our empirical results are encouraging, and show that the local curvature descent improves upon gradient descent.