Objective Priors in the Empirical Bayes Framework

TL;DR

This paper introduces a transformation-invariant non-parametric estimator for priors in Empirical Bayes, balancing minimal informativeness with data-driven insights, addressing overfitting and invariance issues in prior estimation.

Contribution

It proposes a novel, invariant prior estimator that extends reference priors to the empirical Bayes framework, improving robustness and consistency.

Findings

The estimator is transformation invariant.

It provides a natural trade-off between objective and empirical Bayes.

The method addresses overfitting in nonparametric prior estimation.

Abstract

When dealing with Bayesian inference the choice of the prior often remains a debatable question. Empirical Bayes methods offer a data-driven solution to this problem by estimating the prior itself from an ensemble of data. In the nonparametric case, the maximum likelihood estimate is known to overfit the data, an issue that is commonly tackled by regularization. However, the majority of regularizations are ad hoc choices which lack invariance under reparametrization of the model and result in inconsistent estimates for equivalent models. We introduce a non-parametric, transformation invariant estimator for the prior distribution. Being defined in terms of the missing information similar to the reference prior, it can be seen as an extension of the latter to the data-driven setting. This implies a natural interpretation as a trade-off between choosing the least informative prior and incorporating the information provided by the data, a symbiosis between the objective and empirical Bayes methodologies.