The Approximate Duality Gap Technique: A Unified Theory of First-Order Methods

TL;DR

This paper introduces a unified analysis framework for first-order optimization methods based on an approximate duality gap, enabling a comprehensive understanding of their convergence properties across various problem classes.

Contribution

It develops a general technique using an approximate duality gap that unifies the analysis of many first-order methods and characterizes discretization errors for different algorithms.

Findings

Enforces an invariant that the approximate duality gap decreases at a certain rate.

Characterizes discretization errors for various discretization methods.

Extends to multiple problem classes including convex, smooth, and saddle-point problems.

Abstract

We present a general technique for the analysis of first-order methods. The technique relies on the construction of a duality gap for an appropriate approximation of the objective function, where the function approximation improves as the algorithm converges. We show that in continuous time enforcement of an invariant that this approximate duality gap decreases at a certain rate exactly recovers a wide range of first-order continuous-time methods. We characterize the discretization errors incurred by different discretization methods, and show how iteration-complexity-optimal methods for various classes of problems cancel out the discretization error. The techniques are illustrated on various classes of problems -- including convex minimization for Lipschitz-continuous objectives, smooth convex minimization, composite minimization, smooth and strongly convex minimization, solving variational inequalities with monotone operators, and convex-concave saddle-point optimization -- and naturally extend to other settings.