Empirical Bayes Estimators for High-Dimensional Sparse Vectors

TL;DR

This paper introduces an empirical Bayes estimator for high-dimensional sparse vectors, compares it with soft-thresholding, and proposes a hybrid approach that adaptively chooses the better estimator, demonstrating improved performance in compressed sensing applications.

Contribution

The paper develops a new empirical Bayes shrinkage estimator for sparse vectors, analyzes its risk concentration, and introduces a hybrid estimator that adaptively combines it with soft-thresholding for enhanced performance.

Findings

Both estimators' risks concentrate around true risk for large n.

The hybrid estimator's loss approximates the minimum of the two estimators' losses.

Using the estimators in AMP improves compressed sensing denoising performance.

Abstract

The problem of estimating a high-dimensional sparse vector $\boldsymbol{\theta} \in \mathbb{R}^n$ from an observation in i.i.d. Gaussian noise is considered. The performance is measured using squared-error loss. An empirical Bayes shrinkage estimator, derived using a Bernoulli-Gaussian prior, is analyzed and compared with the well-known soft-thresholding estimator. We obtain concentration inequalities for the Stein's unbiased risk estimate and the loss function of both estimators. The results show that for large $n$, both the risk estimate and the loss function concentrate on deterministic values close to the true risk. Depending on the underlying $\boldsymbol{\theta}$, either the proposed empirical Bayes (eBayes) estimator or soft-thresholding may have smaller loss. We consider a hybrid estimator that attempts to pick the better of the soft-thresholding estimator and the eBayes estimator by comparing their risk estimates. It is shown that: i) the loss of the hybrid estimator concentrates on the minimum of the losses of the two competing estimators, and ii) the risk of the hybrid estimator is within order $\frac{1}{\sqrt{n}}$ of the minimum of the two risks. Simulation results are provided to support the theoretical results. Finally, we use the eBayes and hybrid estimators as denoisers in the approximate message passing (AMP) algorithm for compressed sensing, and show that their performance is superior to the soft-thresholding denoiser in a wide range of settings.