New Results on the Polyak Stepsize: Tight Convergence Analysis and Universal Function Classes

TL;DR

This paper provides a tight convergence analysis of the Polyak stepsize in gradient descent, demonstrating its optimality and universality across different function classes, including smooth and H"older conditions.

Contribution

We establish the tightness of PolyakGD's known convergence rates and prove its universal adaptation to various function classes without prior parameter knowledge.

Findings

Convergence rates for smooth strongly convex functions are tight.

PolyakGD exploits floating-point errors to escape worst-case scenarios.

PolyakGD adapts automatically to different function classes.

Abstract

In this paper, we revisit a classical adaptive stepsize strategy for gradient descent: the Polyak stepsize (PolyakGD), originally proposed in Polyak (1969). We study the convergence behavior of PolyakGD from two perspectives: tight worst-case analysis and universality across function classes. As our first main result, we establish the tightness of the known convergence rates of PolyakGD by explicitly constructing worst-case functions. In particular, we show that the $O((1-\frac{1}{\kappa})^K)$ rate for smooth strongly convex functions and the $O(1/K)$ rate for smooth convex functions are both tight. Moreover, we theoretically show that PolyakGD automatically exploits floating-point errors to escape the worst-case behavior. Our second main result provides new convergence guarantees for PolyakGD under both H\"older smoothness and H\"older growth conditions. These findings show that the Polyak stepsize is universal, automatically adapting to various function classes without requiring prior knowledge of problem parameters.