A template for gradient norm minimization

TL;DR

This paper introduces a versatile template for gradient norm minimization applicable to composite optimization problems, including those lacking quadratic growth, providing near-optimal rates and practical adaptivity.

Contribution

It develops a unified, human-readable framework that generalizes existing methods and enables the construction of adaptive, parameter-free algorithms with optimal worst-case guarantees.

Findings

Framework recovers existing approaches via Performance Estimation

Constructs a quasi-online parameter-free method for all composite problems

Preliminary simulations show high practical competitiveness

Abstract

The gradient mapping norm is a strong and easily verifiable stopping criterion for first-order methods on composite problems. When the objective exhibits the quadratic growth property, the gradient mapping norm minimization problem can be solved by online parameter-free and adaptive first-order schemes with near-optimal worst-case rates. In this work we address problems where quadratic growth is absent, a class for which no methods with all the aforementioned properties are known to exist. We formulate a template whose instantiation recovers the existing Performance Estimation derived approaches. Our framework provides a simple human-readable interpretation along with runtime convergence rates for these algorithms. Moreover, our template can be used to construct a quasi-online parameter-free method applicable to the entire class of composite problems while retaining the optimal worst-case rates with the best known proportionality constant. The analysis also allows for adaptivity. Preliminary simulation results suggest that our scheme is highly competitive in practice with the existing approaches, either obtained via Performance Estimation or employing Accumulative Regularization.