Mathematical Theory of Bayesian Statistics for Unknown Information Source

TL;DR

This paper develops a mathematical framework for Bayesian statistics that clarifies properties of model evaluation metrics under unknown or unrealizable data-generating processes, aiding inference when models are inherently uncertain.

Contribution

It introduces a new theoretical foundation for Bayesian inference under unknown uncertainty, clarifying properties of evaluation metrics even with unrealizable models or non-normal posteriors.

Findings

A more precise estimator of generalization loss than leave-one-out CV.

A more accurate approximation of marginal likelihood than BIC.

Optimal hyperparameters differ for generalization loss and marginal likelihood.

Abstract

In statistical inference, uncertainty is unknown and all models are wrong. That is to say, a person who makes a statistical model and a prior distribution is simultaneously aware that both are fictional candidates. To study such cases, statistical measures have been constructed, such as cross validation, information criteria, and marginal likelihood, however, their mathematical properties have not yet been completely clarified when statistical models are under- and over- parametrized. We introduce a place of mathematical theory of Bayesian statistics for unknown uncertainty, which clarifies general properties of cross validation, information criteria, and marginal likelihood, even if an unknown data-generating process is unrealizable by a model or even if the posterior distribution cannot be approximated by any normal distribution. Hence it gives a helpful standpoint for a person who cannot believe in any specific model and prior. This paper consists of three parts. The first is a new result, whereas the second and third are well-known previous results with new experiments. We show there exists a more precise estimator of the generalization loss than leave-one-out cross validation, there exists a more accurate approximation of marginal likelihood than BIC, and the optimal hyperparameters for generalization loss and marginal likelihood are different.