Accelerating Level-Value Adjustment for the Polyak Stepsize

TL;DR

This paper introduces an efficient method for dynamically estimating the optimal value in Polyak stepsize algorithms, ensuring convergence and improving practical performance in non-smooth convex optimization.

Contribution

It proposes a novel level adjustment technique using the Polyak Stepsize Violation Detector to guarantee convergence without prior knowledge of the optimal value.

Findings

The method guarantees convergence of both level and objective function values.

Empirical results show improved efficiency over existing approaches.

The approach simplifies subgradient calculations while maintaining accuracy.

Abstract

The Polyak stepsize has been widely used in subgradient methods for non-smooth convex optimization. However, calculating the stepsize requires the optimal value, which is generally unknown. Therefore, dynamic estimations of the optimal value are usually needed. In this paper, to guarantee convergence, a series of level values is constructed to estimate the optimal value successively. This is achieved by developing a decision-guided procedure that involves solving a novel, easy-to-solve linear constraint satisfaction problem referred to as the ``Polyak Stepsize Violation Detector'' (PSVD). Once a violation is detected, the level value is recalculated. We rigorously establish the convergence for both the level values and the objective function values. Furthermore, with our level adjustment approach, calculating an approximate subgradient in each iteration is sufficient for convergence. A series of empirical tests of convex optimization problems with diverse characteristics demonstrates the practical advantages of our approach over existing methods.