History and Nature of the Jeffreys-Lindley Paradox

TL;DR

This paper revisits the Jeffreys-Lindley paradox, clarifying its historical development, role of priors, and demonstrating its generality across Bayesian and frequentist frameworks, challenging common misconceptions.

Contribution

It provides a comprehensive historical and conceptual analysis of the Jeffreys-Lindley paradox, emphasizing the importance of priors and showing its relevance beyond point hypotheses.

Findings

The paradox is rooted in Jeffreys's early work and the role of the $\sqrt{n}$ scaling.

Prior distributions implicitly correct for selection effects.

The paradox is applicable in both Bayesian and frequentist contexts.

Abstract

The Jeffreys-Lindley paradox exposes a rift between Bayesian and frequentist hypothesis testing that strikes at the heart of statistical inference. Contrary to what most current literature suggests, the paradox was central to the Bayesian testing methodology developed by Sir Harold Jeffreys in the late 1930s. Jeffreys showed that the evidence against a point-null hypothesis $\mathcal{H}_0$ scales with $\sqrt{n}$ and repeatedly argued that it would therefore be mistaken to set a threshold for rejecting $\mathcal{H}_0$ at a constant multiple of the standard error. Here we summarize Jeffreys's early work on the paradox and clarify his reasons for including the $\sqrt{n}$ term. The prior distribution is seen to play a crucial role; by implicitly correcting for selection, small parameter values are identified as relatively surprising under $\mathcal{H}_1$. We highlight the general nature of the paradox by presenting both a fully frequentist and a fully Bayesian version. We also demonstrate that the paradox does not depend on assigning prior mass to a point hypothesis, as is commonly believed.