MMSE Approximation For Sparse Coding Algorithms Using Stochastic Resonance

TL;DR

This paper introduces a novel approach to improve sparse coding algorithms by leveraging stochastic resonance, adding controlled noise to approximate the MMSE estimator efficiently, with empirical and theoretical validation.

Contribution

The paper proposes a new stochastic resonance-based method to approximate the MMSE estimator in sparse coding, enhancing performance and computational efficiency.

Findings

The proposed methods effectively approximate the MMSE estimator.

Both variants outperform traditional sparse coding in experiments.

Theoretical analysis confirms convergence and robustness.

Abstract

Sparse coding refers to the pursuit of the sparsest representation of a signal in a typically overcomplete dictionary. From a Bayesian perspective, sparse coding provides a Maximum a Posteriori (MAP) estimate of the unknown vector under a sparse prior. In this work, we suggest enhancing the performance of sparse coding algorithms by a deliberate and controlled contamination of the input with random noise, a phenomenon known as stochastic resonance. The proposed method adds controlled noise to the input and estimates a sparse representation from the perturbed signal. A set of such solutions is then obtained by projecting the original input signal onto the recovered set of supports. We present two variants of the described method, which differ in their final step. The first is a provably convergent approximation to the Minimum Mean Square Error (MMSE) estimator, relying on the generative model and applying a weighted average over the recovered solutions. The second is a relaxed variant of the former that simply applies an empirical mean. We show that both methods provide a computationally efficient approximation to the MMSE estimator, which is typically intractable to compute. We demonstrate our findings empirically and provide a theoretical analysis of our method under several different cases.