Perturbed Fenchel Duality and Primal-Dual Convergence of First-Order Methods

TL;DR

This paper demonstrates that many first-order optimization methods satisfy a perturbed Fenchel duality, enabling unified convergence analysis and introducing a new bundle method for saddle problems.

Contribution

It extends the perturbed Fenchel duality framework to more first-order methods and proposes a novel single-cut bundle method with proven convergence.

Findings

Primal-dual convergence of dual averaging and bundle methods.

Unified convergence proof via perturbed Fenchel duality.

Introduction of a new bundle method for saddle problems.

Abstract

It has been shown that many first-order methods satisfy the perturbed Fenchel duality inequality, which yields a unified derivation of convergence. More first-order methods are discussed in this paper, e.g., dual averaging and bundle method. We show primal-dual convergence of them on convex optimization by proving the perturbed Fenchel duality property. We also propose a single-cut bundle method for saddle problem, and prove its convergence in a similar manner.