H-invariance theory: A complete characterization of minimax optimal fixed-point algorithms

TL;DR

This paper characterizes all fixed-point algorithms with predetermined step-sizes that achieve minimax optimal convergence rates, using a novel H-invariance theory based on polynomial invariants and certificates.

Contribution

It introduces a complete characterization of optimal fixed-point algorithms through H-invariants and H-certificates, providing a new mathematical framework for understanding acceleration.

Findings

All optimal algorithms can be described by H-matrices with constant invariants.

H-invariants remain constant across all optimal algorithms, defining their common structure.

Nonnegativity of H-certificates precisely identifies the region of optimality.

Abstract

For nonexpansive fixed-point problems, Halpern's method with optimal parameters, its so-called H-dual algorithm, and in fact, an infinite family of algorithms containing them, all exhibit the exactly minimax optimal convergence rates. In this work, we provide a characterization of the complete, exhaustive family of distinct algorithms using predetermined step-sizes, represented as lower triangular H-matrices, which attain the same optimal convergence rate. The characterization is based on polynomials in the entries of the H-matrix that we call H-invariants, whose values stay constant over all optimal H-matrices, together with H-certificates, of which nonnegativity precisely specifies the region of optimality within the common level set of H-invariants. The H-invariance theory we present offers a novel view of optimal acceleration in first-order optimization as a mathematical study of carefully selected invariants, certificates, and structures induced by them.