A proof of consistency and model-selection optimality on the empirical Bayes method

TL;DR

This paper proves the consistency and optimality of the empirical Bayes method using MMLE for hyperparameter inference in large models, especially within the exponential family, ensuring reliable model selection and estimation.

Contribution

It provides the first rigorous proof of the consistency and model-selection optimality of MMLE in empirical Bayes methods for exponential family models.

Findings

MMLE is consistent for variance hyperparameters in linear models.

MMLE asymptotically minimizes KL divergence between prior and true distribution.

Results hold even when the true distribution is outside the model space.

Abstract

We study the consistency and optimality of the maximum marginal likelihood estimate (MMLE) in the hyperparameter inference for large-degree-of-freedom models. We perform main analyses within the exponential family, where the natural parameters are hyperparameters. First, we prove the consistency of the MMLE for the general linear models when estimating the scales of variance in the likelihood and prior. The proof is independent of the number ratio of data to model parameters and excepts the ill-posedness of the associated regularized least-square model-parameter estimate that is shown asymptotically unbiased. Second, we generalize the proof to other models with a finite number of hyperparameters. We find that the extensive properties of cost functions in the exponential family generally yield the consistency of the MMLE for the likelihood hyperparameters. Besides, we show the MMLE asymptotically almost surely minimizes the Kullback-Leibler divergence between the prior and true predictive distributions even if the true data distribution is outside the model space under the hypothetical asymptotic normality of the predictive distributions applicable to non-exponential model families. Our proof validates the empirical Bayes method using the hyperparameter MMLE in the asymptotics of many model parameters, ensuring the same qualification for the empirical-cross-entropy cross-validation.