Time-Reversed Dissipation Induces Duality Between Minimizing Gradient Norm and Function Value

TL;DR

This paper introduces H-duality, a novel concept linking methods that minimize function values and gradient magnitudes in convex optimization, revealing new symmetries and leading to improved algorithms.

Contribution

It presents H-duality as a new form of duality in optimization, providing insights into method symmetries and developing faster, simpler algorithms for convex minimization.

Findings

H-duality reveals a one-to-one correspondence between minimizing function values and gradient magnitudes.

The paper derives a new class of methods that efficiently reduce gradient magnitudes.

A new composite minimization method surpasses FISTA-G in simplicity and speed.

Abstract

In convex optimization, first-order optimization methods efficiently minimizing function values have been a central subject study since Nesterov's seminal work of 1983. Recently, however, Kim and Fessler's OGM-G and Lee et al.'s FISTA-G have been presented as alternatives that efficiently minimize the gradient magnitude instead. In this paper, we present H-duality, which represents a surprising one-to-one correspondence between methods efficiently minimizing function values and methods efficiently minimizing gradient magnitude. In continuous-time formulations, H-duality corresponds to reversing the time dependence of the dissipation/friction term. To the best of our knowledge, H-duality is different from Lagrange/Fenchel duality and is distinct from any previously known duality or symmetry relations. Using H-duality, we obtain a clearer understanding of the symmetry between Nesterov's method and OGM-G, derive a new class of methods efficiently reducing gradient magnitudes of smooth convex functions, and find a new composite minimization method that is simpler and faster than FISTA-G.