Are good local minima wide in sparse recovery?

TL;DR

This paper investigates the properties of local minima in sparse recovery, showing that perturbations and deep-learning-inspired methods can significantly improve the performance of iterative hard thresholding algorithms.

Contribution

It introduces noise perturbations and a deep-learning framework to enhance the effectiveness of IHT in sparse recovery tasks.

Findings

Perturbing IHT with noise improves recovery accuracy.

Dropout-based perturbations outperform classical IHT.

Achieves 3 to 6 times lower average objective errors.

Abstract

The idea of compressed sensing is to exploit representations in suitable (overcomplete) dictionaries that allow to recover signals far beyond the Nyquist rate provided that they admit a sparse representation in the respective dictionary. The latter gives rise to the sparse recovery problem of finding the best sparse linear approximation of given data in a given generating system. In this paper we analyze the iterative hard thresholding (IHT) algorithm as one of the most popular greedy methods for solving the sparse recovery problem, and demonstrate that systematically perturbing the IHT algorithm by adding noise to intermediate iterates yields improved results. Further improvements can be obtained by entirely rephrasing the problem as a parametric deep-learning-type of optimization problem. By introducing perturbations via dropout, we demonstrate to significantly outperform the classical IHT algorithm, obtaining $3$ to $6$ times lower average objective errors.