Duality between subgradient and conditional gradient methods

TL;DR

This paper reveals a fundamental equivalence between mirror descent and conditional gradient algorithms via convex duality, impacting optimization methods in machine learning with non-smooth components.

Contribution

It establishes a formal duality-based connection between subgradient and conditional gradient methods, providing new insights and convergence guarantees.

Findings

Primal subgradient and dual conditional gradient methods are equivalent for certain problems.

Dual interpretation enables line search techniques for mirror descent.

Provides convergence guarantees for primal-dual certificates.

Abstract

Given a convex optimization problem and its dual, there are many possible first-order algorithms. In this paper, we show the equivalence between mirror descent algorithms and algorithms generalizing the conditional gradient method. This is done through convex duality, and implies notably that for certain problems, such as for supervised machine learning problems with non-smooth losses or problems regularized by non-smooth regularizers, the primal subgradient method and the dual conditional gradient method are formally equivalent. The dual interpretation leads to a form of line search for mirror descent, as well as guarantees of convergence for primal-dual certificates.