A Unified Optimization View on Generalized Matching Pursuit and Frank-Wolfe

TL;DR

This paper unifies the analysis of matching pursuit and Frank-Wolfe algorithms, providing explicit convergence rates for both, applicable to general sets of atoms without incoherence or sparsity assumptions.

Contribution

It offers the first explicit optimization convergence rates for matching pursuit methods, unifying them with Frank-Wolfe and establishing affine invariance.

Findings

Sublinear ($1/t$) convergence for smooth objectives

Linear convergence for strongly convex objectives

Algorithms are affine invariant and assumption-free

Abstract

Two of the most fundamental prototypes of greedy optimization are the matching pursuit and Frank-Wolfe algorithms. In this paper, we take a unified view on both classes of methods, leading to the first explicit convergence rates of matching pursuit methods in an optimization sense, for general sets of atoms. We derive sublinear ($1/t$) convergence for both classes on general smooth objectives, and linear convergence on strongly convex objectives, as well as a clear correspondence of algorithm variants. Our presented algorithms and rates are affine invariant, and do not need any incoherence or sparsity assumptions.