Far from Asymptopia

TL;DR

This paper critiques Jeffreys prior in Bayesian inference for complex models, showing it introduces bias due to misaligned treatment of relevant parameters, and proposes a data-dependent measure for unbiased inference.

Contribution

It identifies the bias introduced by Jeffreys prior in models with smaller effective dimensionality and proposes a principled, data-dependent measure that reduces this bias.

Findings

Jeffreys prior causes significant bias in typical scientific models.

The proposed measure depends on the amount of data and becomes similar to Jeffreys prior asymptotically.

Unbiased inference requires an impractically large amount of data, exponential in model complexity.

Abstract

Inference from limited data requires a notion of measure on parameter space, most explicit in the Bayesian framework as a prior. Here we demonstrate that Jeffreys prior, the best-known uninformative choice, introduces enormous bias when applied to typical scientific models. Such models have a relevant effective dimensionality much smaller than the number of microscopic parameters. Because Jeffreys prior treats all microscopic parameters equally, it is from uniform when projected onto the sub-space of relevant parameters, due to variations in the local co-volume of irrelevant directions. We present results on a principled choice of measure which avoids this issue, leading to unbiased inference in complex models. This optimal prior depends on the quantity of data to be gathered, and approaches Jeffreys prior in the asymptotic limit. However, this limit cannot be justified without an impossibly large amount of data, exponential in the number of microscopic parameters.