Bayes and empirical Bayes: do they merge?

TL;DR

This paper investigates the relationship between Bayesian and empirical Bayes methods, providing rigorous conditions under which their posterior distributions merge, especially in large samples, with applications to density estimation.

Contribution

It offers a formal justification for empirical Bayes via merging concepts and establishes conditions for consistency and asymptotic hyperparameter selection.

Findings

Empirical Bayes posteriors can merge with Bayesian posteriors under certain conditions.

Conditions for the consistency of empirical Bayes procedures are provided.

Empirical Bayes asymptotically identifies hyperparameters that favor the true model.

Abstract

Bayesian inference is attractive for its coherence and good frequentist properties. However, it is a common experience that eliciting a honest prior may be difficult and, in practice, people often take an {\em empirical Bayes} approach, plugging empirical estimates of the prior hyperparameters into the posterior distribution. Even if not rigorously justified, the underlying idea is that, when the sample size is large, empirical Bayes leads to "similar" inferential answers. Yet, precise mathematical results seem to be missing. In this work, we give a more rigorous justification in terms of merging of Bayes and empirical Bayes posterior distributions. We consider two notions of merging: Bayesian weak merging and frequentist merging in total variation. Since weak merging is related to consistency, we provide sufficient conditions for consistency of empirical Bayes posteriors. Also, we show that, under regularity conditions, the empirical Bayes procedure asymptotically selects the value of the hyperparameter for which the prior mostly favors the "truth". Examples include empirical Bayes density estimation with Dirichlet process mixtures.