Sparse approximation problem: how rapid simulated annealing succeeds and fails

TL;DR

This paper investigates the effectiveness of simulated annealing for sparse approximation, demonstrating its success in noiseless scenarios with planted solutions and identifying failure modes related to phase transitions and high noise levels.

Contribution

The study shows that simulated annealing can efficiently find planted sparse solutions in certain conditions, even outperforming traditional $ ext{l}_1$-relaxation methods, and analyzes its limitations.

Findings

Simulated annealing finds planted solutions rapidly in noiseless cases.

Failure occurs near phase transition points where metastable states emerge.

In high-noise scenarios, the algorithm quickly finds solutions with minimal distortion.

Abstract

Information processing techniques based on sparseness have been actively studied in several disciplines. Among them, a mathematical framework to approximately express a given dataset by a combination of a small number of basis vectors of an overcomplete basis is termed the {\em sparse approximation}. In this paper, we apply simulated annealing, a metaheuristic algorithm for general optimization problems, to sparse approximation in the situation where the given data have a planted sparse representation and noise is present. The result in the noiseless case shows that our simulated annealing works well in a reasonable parameter region: the planted solution is found fairly rapidly. This is true even in the case where a common relaxation of the sparse approximation problem, the $\ell_1$-relaxation, is ineffective. On the other hand, when the dimensionality of the data is close to the number of non-zero components, another metastable state emerges, and our algorithm fails to find the planted solution. This phenomenon is associated with a first-order phase transition. In the case of very strong noise, it is no longer meaningful to search for the planted solution. In this situation, our algorithm determines a solution with close-to-minimum distortion fairly quickly.