Parametric Mean-Field empirical Bayes in high-dimensional linear regression

TL;DR

This paper studies the asymptotic behavior of parametric empirical Bayes estimators in high-dimensional linear regression, revealing phase transitions and proposing methods for improved inference in different regimes.

Contribution

It characterizes the phase transition for the variational empirical Bayes estimator and introduces a debiasing technique for high-dimensional settings beyond the phase transition.

Findings

vEB estimator is optimal up to p=o(n^{2/3})

Calibration enables valid inference in the first regime

Debiasing improves performance beyond p=o(n^{2/3})

Abstract

In this paper, we consider the problem of parametric empirical Bayes estimation of an i.i.d. prior in high-dimensional Bayesian linear regression, with random design. We obtain the asymptotic distribution of the variational Empirical Bayes (vEB) estimator, which approximately maximizes a variational lower bound of the intractable marginal likelihood. We characterize a sharp phase transition behavior for the vEB estimator -- namely that it is information theoretically optimal (in terms of limiting variance) up to $p=o(n^{2/3})$ while it suffers from a sub-optimal convergence rate in higher dimensions. In the first regime, i.e., when $p=o(n^{2/3})$, we show how the estimated prior can be calibrated to enable valid coordinate-wise and delocalized inference, both under the \emph{empirical Bayes posterior} and the oracle posterior. In the second regime, we propose a debiasing technique as a way to improve the performance of the vEB estimator beyond $p=o(n^{2/3})$. Extensive numerical experiments corroborate our theoretical findings.