Convergence of first-order methods via the convex conjugate

TL;DR

This paper presents a unified approach to analyze the convergence rates of various first-order optimization methods for convex minimization, using convex conjugates to simplify proofs.

Contribution

It introduces a generic bound based on convex conjugates that unifies the convergence analysis of subgradient, gradient, and accelerated gradient methods.

Findings

Unified proof framework for convergence rates

Simplifies analysis of first-order methods

Applicable to various convex optimization algorithms

Abstract

This paper gives a unified and succinct approach to the $O(1/\sqrt{k}), O(1/k),$ and $O(1/k^2)$ convergence rates of the subgradient, gradient, and accelerated gradient methods for unconstrained convex minimization. In the three cases the proof of convergence follows from a generic bound defined by the convex conjugate of the objective function.