The Bayes Factor Reversal Paradox

TL;DR

This paper uncovers a paradox within Bayesian inference where, in normal models, the same data can lead to opposite conclusions depending solely on prior variance choices, highlighting a significant arbitrariness in Bayesian hypothesis testing.

Contribution

It demonstrates a new paradox showing that Bayes factors can reverse evidence conclusions based on prior variance, even with realistic sample sizes.

Findings

Bayes factors can indicate opposite evidence for the same data.

The paradox occurs at common significance levels like 0.05.

It highlights prior scale choice as a critical and arbitrary factor.

Abstract

In 1957, Lindley published "A statistical paradox" in Biometrika, revealing a fundamental conflict between frequentist and Bayesian inference as sample size approaches infinity. We present a new paradox of a different kind: a conflict within Bayesian inference itself. In the normal model with known variance, we prove that for any two-sided statistically significant result at the 0.05 level there exist prior variances such that the Bayes factor indicates evidence for the alternative with one choice while indicating evidence for the null with another. Thus, the same data, testing the same hypothesis, can yield opposite conclusions depending solely on prior choice. This answers Robert's 2016 call to investigate the impact of the prior scale on Bayes factors and formalises his concern that this choice involves arbitrariness to a high degree. Unlike the Jeffreys-Lindley paradox, which requires sample size approaching infinity, the paradox we identify occurs with realistic sample sizes.