Online Orthogonal Matching Pursuit

TL;DR

This paper introduces Online Orthogonal Matching Pursuit, an online algorithm for support recovery in sparse linear regression that balances sample efficiency and computational complexity with proven guarantees.

Contribution

The paper proposes a novel online greedy algorithm for feature selection in sparse linear models, with theoretical analysis of its guarantees and computational complexity.

Findings

Algorithm achieves support recovery with theoretical guarantees.

Computational complexity is analyzed and shown to be efficient.

Supports online feature selection in high-dimensional settings.

Abstract

Greedy algorithms for feature selection are widely used for recovering sparse high-dimensional vectors in linear models. In classical procedures, the main emphasis was put on the sample complexity, with little or no consideration of the computation resources required. We present a novel online algorithm: Online Orthogonal Matching Pursuit (OOMP) for online support recovery in the random design setting of sparse linear regression. Our procedure selects features sequentially, alternating between allocation of samples only as needed to candidate features, and optimization over the selected set of variables to estimate the regression coefficients. Theoretical guarantees about the output of this algorithm are proven and its computational complexity is analysed.