Convergence Rates of Empirical Bayes Posterior Distributions: A Variational Perspective

TL;DR

This paper establishes convergence rates for empirical Bayes posteriors in high-dimensional and nonparametric settings by connecting them to variational Bayes and introducing a new prior decomposition technique.

Contribution

It reveals a variational interpretation of empirical Bayes posteriors and develops a novel prior decomposition method for non-discrete hyperparameter sets.

Findings

Empirical Bayes posteriors can be viewed as variational approximations.

The new prior decomposition technique handles continuous hyperparameter sets.

Convergence rates are derived for density estimation and sparse regression.

Abstract

We study the convergence rates of empirical Bayes posterior distributions for nonparametric and high-dimensional inference. We show that as long as the hyperparameter set is discrete, the empirical Bayes posterior distribution induced by the maximum marginal likelihood estimator can be regarded as a variational approximation to a hierarchical Bayes posterior distribution. This connection between empirical Bayes and variational Bayes allows us to leverage the recent results in the variational Bayes literature, and directly obtains the convergence rates of empirical Bayes posterior distributions from a variational perspective. For a more general hyperparameter set that is not necessarily discrete, we introduce a new technique called "prior decomposition" to deal with prior distributions that can be written as convex combinations of probability measures whose supports are low-dimensional subspaces. This leads to generalized versions of the classical "prior mass and testing" conditions for the convergence rates of empirical Bayes. Our theory is applied to a number of statistical estimation problems including nonparametric density estimation and sparse linear regression.