A Mean Field Approach to Empirical Bayes Estimation in High-dimensional Linear Regression

TL;DR

This paper develops a computationally efficient empirical Bayes method for high-dimensional linear regression, establishing theoretical guarantees and enabling Bayesian inference without assuming sparsity.

Contribution

It introduces a variational empirical Bayes approach with proven asymptotic consistency and accuracy, applicable to both deterministic and random designs with feature correlations.

Findings

Asymptotic consistency of NPMLE and variational surrogate.

Accurate approximation of the oracle posterior under Wasserstein metric.

Feasible Bayesian inference with credible intervals and optimal estimation.

Abstract

We study empirical Bayes estimation in high-dimensional linear regression. To facilitate computationally efficient estimation of the underlying prior, we adopt a variational empirical Bayes approach, introduced originally in Carbonetto and Stephens (2012) and Kim et al. (2022). We establish asymptotic consistency of the nonparametric maximum likelihood estimator (NPMLE) and its (computable) naive mean field variational surrogate under mild assumptions on the design and the prior. Assuming, in addition, that the naive mean field approximation has a dominant optimizer, we develop a computationally efficient approximation to the oracle posterior distribution, and establish its accuracy under the 1-Wasserstein metric. This enables computationally feasible Bayesian inference; e.g., construction of posterior credible intervals with an average coverage guarantee, Bayes optimal estimation for the regression coefficients, estimation of the proportion of non-nulls, etc. Our analysis covers both deterministic and random designs, and accommodates correlations among the features. To the best of our knowledge, this provides the first rigorous nonparametric empirical Bayes method in a high-dimensional regression setting without sparsity.