Can Bayes Factors "Prove" the Null Hypothesis?

TL;DR

This paper examines the limitations of Bayes Factors in conclusively proving the null hypothesis, especially with large samples, highlighting scenarios where BF can be misleading due to ignored hypotheses.

Contribution

It identifies conditions causing Bayes Factors to be inconclusive or misleading and explores methods to address these issues in Bayesian hypothesis testing.

Findings

Large sample sizes can lead to conflicting Bayes Factors favoring null and alternative hypotheses.

Bayes Factors may favor the null hypothesis even when other hypotheses are strongly supported.

The paper discusses conditions and methods to resolve the interpretational issues of Bayes Factors.

Abstract

It is possible to obtain a large Bayes Factor (BF) favoring the null hypothesis when both the null and alternative hypotheses have low likelihoods, and there are other hypotheses being ignored that are much more strongly supported by the data. As sample sizes become large it becomes increasingly probable that a strong BF favouring a point null against a conventional Bayesian vague alternative co-occurs with a BF favouring various specific alternatives against the null. For any BF threshold q and sample mean, there is a value n such that sample sizes larger than n guarantee that although the BF comparing H0 against a conventional (vague) alternative exceeds q, nevertheless for some range of hypothetical {\mu}, a BF comparing H0 against {\mu} in that range falls below 1/q. This paper discusses the conditions under which this conundrum occurs and investigates methods for resolving it.