On Matching Pursuit and Coordinate Descent

TL;DR

This paper unifies the analysis of coordinate descent and matching pursuit algorithms, establishing new convergence rates including the first accelerated rate for both methods on convex objectives.

Contribution

It provides a unified, affine invariant analysis of both algorithms, deriving the tightest known rates for steepest coordinate descent and introducing the first accelerated convergence for matching pursuit.

Findings

Affine invariant sublinear $ ext{O}(1/t)$ rates on smooth objectives

Linear convergence on strongly convex objectives

First accelerated $ ext{O}(1/t^2)$ rate for matching pursuit and steepest coordinate descent

Abstract

Two popular examples of first-order optimization methods over linear spaces are coordinate descent and matching pursuit algorithms, with their randomized variants. While the former targets the optimization by moving along coordinates, the latter considers a generalized notion of directions. Exploiting the connection between the two algorithms, we present a unified analysis of both, providing affine invariant sublinear $\mathcal{O}(1/t)$ rates on smooth objectives and linear convergence on strongly convex objectives. As a byproduct of our affine invariant analysis of matching pursuit, our rates for steepest coordinate descent are the tightest known. Furthermore, we show the first accelerated convergence rate $\mathcal{O}(1/t^2)$ for matching pursuit and steepest coordinate descent on convex objectives.