Maximal Sparsity with Deep Networks?

TL;DR

This paper explores how deep neural networks can be trained to improve sparse signal recovery, especially in challenging scenarios with coherent dictionaries, and demonstrates their effectiveness in a practical photometric stereo application.

Contribution

It shows both theoretically and empirically that learned deep networks can recover maximally sparse representations where traditional algorithms fail, particularly with coherent dictionaries.

Findings

Deep networks can outperform traditional sparse estimation methods in high coherence regimes.

Trained models successfully recover sparse signals in practical photometric stereo tasks.

The approach enhances estimation accuracy in challenging sparse recovery scenarios.

Abstract

The iterations of many sparse estimation algorithms are comprised of a fixed linear filter cascaded with a thresholding nonlinearity, which collectively resemble a typical neural network layer. Consequently, a lengthy sequence of algorithm iterations can be viewed as a deep network with shared, hand-crafted layer weights. It is therefore quite natural to examine the degree to which a learned network model might act as a viable surrogate for traditional sparse estimation in domains where ample training data is available. While the possibility of a reduced computational budget is readily apparent when a ceiling is imposed on the number of layers, our work primarily focuses on estimation accuracy. In particular, it is well-known that when a signal dictionary has coherent columns, as quantified by a large RIP constant, then most tractable iterative algorithms are unable to find maximally sparse representations. In contrast, we demonstrate both theoretically and empirically the potential for a trained deep network to recover minimal $\ell_0$-norm representations in regimes where existing methods fail. The resulting system is deployed on a practical photometric stereo estimation problem, where the goal is to remove sparse outliers that can disrupt the estimation of surface normals from a 3D scene.