Linear Convergence of Stochastic Iterative Greedy Algorithms with Sparse Constraints

TL;DR

This paper introduces stochastic greedy algorithms with proven linear convergence for non-convex sparse optimization problems, demonstrating robustness to approximation errors and superior performance over deterministic methods.

Contribution

The paper develops stochastic variants of greedy algorithms for non-convex sparse problems, providing convergence guarantees and robustness to inexact computations.

Findings

Algorithms achieve linear convergence in expectation.

Methods are robust to approximate gradients and projections.

Numerical experiments confirm theoretical results and outperform deterministic methods.

Abstract

Motivated by recent work on stochastic gradient descent methods, we develop two stochastic variants of greedy algorithms for possibly non-convex optimization problems with sparsity constraints. We prove linear convergence in expectation to the solution within a specified tolerance. This generalized framework applies to problems such as sparse signal recovery in compressed sensing, low-rank matrix recovery, and covariance matrix estimation, giving methods with provable convergence guarantees that often outperform their deterministic counterparts. We also analyze the settings where gradients and projections can only be computed approximately, and prove the methods are robust to these approximations. We include many numerical experiments which align with the theoretical analysis and demonstrate these improvements in several different settings.