The Use of Mutual Coherence to Prove $\ell^1/\ell^0$-Equivalence in Classification Problems

TL;DR

This paper explores the conditions under which $ ext{l}^1$-minimization reliably recovers sparse solutions in classification tasks, revealing that class separation and approximate equivalence are crucial for success.

Contribution

It proves that deterministic checks for $ ext{l}^1/ ext{l}^0$-equivalence conflict with high coherence within classes and shows that class separation and nonlinear transforms enable effective sparse recovery.

Findings

High coherence within classes conflicts with $ ext{l}^1/ ext{l}^0$-equivalence verification.

Class separation improves the ability of $ ext{l}^1$-minimization to recover sparse solutions.

Approximate equivalence, facilitated by nonlinear transforms, is vital for successful classification.

Abstract

We consider the decomposition of a signal over an overcomplete set of vectors. Minimization of the $\ell^1$-norm of the coefficient vector can often retrieve the sparsest solution (so-called "$\ell^1/\ell^0$-equivalence"), a generally NP-hard task, and this fact has powered the field of compressed sensing. Wright et al.'s sparse representation-based classification (SRC) applies this relationship to machine learning, wherein the signal to be decomposed represents the test sample and columns of the dictionary are training samples. We investigate the relationships between $\ell^1$-minimization, sparsity, and classification accuracy in SRC. After proving that the tractable, deterministic approach to verifying $\ell^1/\ell^0$-equivalence fundamentally conflicts with the high coherence between same-class training samples, we demonstrate that $\ell^1$-minimization can still recover the sparsest solution when the classes are well-separated. Further, using a nonlinear transform so that sparse recovery conditions may be satisfied, we demonstrate that approximate (not strict) equivalence is key to the success of SRC.