Finding a sparse vector in a subspace: Linear sparsity using alternating directions

TL;DR

This paper introduces a nonconvex alternating directions algorithm that efficiently finds sparse vectors in subspaces, outperforming convex heuristics especially when the sparsity level is high, with applications in signal processing and machine learning.

Contribution

The paper presents the first practical algorithm achieving linear scaling for sparse vector recovery in subspaces under the planted sparse model, surpassing convex methods.

Findings

The proposed nonconvex method succeeds with high probability even when the sparsity fraction is constant.

Convex heuristics fail when the nonzero entries exceed O(1/√n), but the new approach does not.

Empirical results show effectiveness in sparse dictionary learning and other challenging models.

Abstract

Is it possible to find the sparsest vector (direction) in a generic subspace $\mathcal{S} \subseteq \mathbb{R}^p$ with $\mathrm{dim}(\mathcal{S})= n < p$? This problem can be considered a homogeneous variant of the sparse recovery problem, and finds connections to sparse dictionary learning, sparse PCA, and many other problems in signal processing and machine learning. In this paper, we focus on a **planted sparse model** for the subspace: the target sparse vector is embedded in an otherwise random subspace. Simple convex heuristics for this planted recovery problem provably break down when the fraction of nonzero entries in the target sparse vector substantially exceeds $O(1/\sqrt{n})$. In contrast, we exhibit a relatively simple nonconvex approach based on alternating directions, which provably succeeds even when the fraction of nonzero entries is $\Omega(1)$. To the best of our knowledge, this is the first practical algorithm to achieve linear scaling under the planted sparse model. Empirically, our proposed algorithm also succeeds in more challenging data models, e.g., sparse dictionary learning.