Sharp Convergence Rates for Matching Pursuit

TL;DR

This paper establishes the precise convergence rate of matching pursuit algorithms, demonstrating that their decay rate is suboptimal compared to other greedy methods, and provides a worst-case dictionary construction to prove this.

Contribution

It provides a sharp characterization of the convergence rate for matching pursuit, closing the gap between existing bounds and identifying its suboptimality.

Findings

Matching pursuit converges at rate n^(-α) with α ≈ 0.182.

A worst-case dictionary demonstrates the tightness of the upper bound.

Matching pursuit's rate is suboptimal compared to other greedy algorithms.

Abstract

We study the fundamental limits of matching pursuit, or the pure greedy algorithm, for approximating a target function $ f $ by a linear combination $f_n$ of $n$ elements from a dictionary. When the target function is contained in the variation space corresponding to the dictionary, many impressive works over the past few decades have obtained upper and lower bounds on the error $\|f-f_n\|$ of matching pursuit, but they do not match. The main contribution of this paper is to close this gap and obtain a sharp characterization of the decay rate, $n^{-\alpha}$, of matching pursuit. Specifically, we construct a worst case dictionary which shows that the existing best upper bound cannot be significantly improved. It turns out that, unlike other greedy algorithm variants which converge at the optimal rate $ n^{-1/2}$, the convergence rate $n^{-\alpha}$ is suboptimal. Here, $\alpha \approx 0.182$ is determined by the solution to a certain non-linear equation.