MLE convergence speed to information projection of exponential family: Criterion for model dimension and sample size -- complete proof version--

TL;DR

This paper develops a criterion based on asymptotic risk expansion to determine if the MLE is sufficiently close to the information projection in exponential family models, aiding in model acceptance and sample size decisions.

Contribution

It introduces the $p-n$ criterion for assessing MLE convergence to the information projection, applicable even for complex models without explicit normalization constants.

Findings

Derived asymptotic expansion of estimation risk up to order $n^{-2}$

Proposed the $p-n$ criterion for model and sample adequacy

Demonstrated the criterion on practical datasets

Abstract

For a parametric model of distributions, the closest distribution in the model to the true distribution located outside the model is considered. Measuring the closeness between two distributions with the Kullback-Leibler (K-L) divergence, the closest distribution is called the "information projection." The estimation risk of the maximum likelihood estimator (MLE) is defined as the expectation of K-L divergence between the information projection and the predictive distribution with plugged-in MLE. Here, the asymptotic expansion of the risk is derived up to $n^{-2}$-order, and the sufficient condition on the risk for the Bayes error rate between the true distribution and the information projection to be lower than a specified value is investigated. Combining these results, the "$p-n$ criterion" is proposed, which determines whether the MLE is sufficiently close to the information projection for the given model and sample. In particular, the criterion for an exponential family model is relatively simple and can be used for a complex model with no explicit form of normalizing constant. This criterion can constitute a solution to the sample size or model acceptance problem. Use of the $p-n$ criteria is demonstrated for two practical datasets. The relationship between the results and information criteria is also studied.