Statistical Inference and Exact Saddle Point Approximations

TL;DR

This paper explores when Bayesian, frequentist, and MDL inference methods align, revealing that for exponential families, this occurs only under specific conditions related to saddle point approximations, with exact cases identified.

Contribution

It establishes the precise conditions under which Bayesian, frequentist, and MDL inferences coincide for exponential families, highlighting the role of exact saddle point approximations.

Findings

Exact saddle point approximation occurs only for specific exponential families.

Gaussian, Gamma, and inverse Gaussian families are uniquely characterized by this property.

In higher dimensions, more exponential families exhibit these exact approximations.

Abstract

Statistical inference may follow a frequentist approach or it may follow a Bayesian approach or it may use the minimum description length principle (MDL). Our goal is to identify situations in which these different approaches to statistical inference coincide. It is proved that for exponential families MDL and Bayesian inference coincide if and only if the renormalized saddle point approximation for the conjugated exponential family is exact. For 1-dimensional exponential families the only families with exact renormalized saddle point approximations are the Gaussian location family, the Gamma family and the inverse Gaussian family. They are conjugated families of the Gaussian location family, the Gamma family and the Poisson-exponential family. The first two families are self-conjugated implying that only for the two first families the Bayesian approach is consistent with the frequentist approach. In higher dimensions there are more examples.