Empirical Bayes factors for common hypothesis tests

TL;DR

This paper introduces empirical Bayes factors for hypothesis testing that address prior knowledge issues, relate to information criteria, and offer an objective, interpretable scale for statistical evidence.

Contribution

It revisits the posterior Bayes factor, proposes bias adjustments, develops test-based empirical Bayes factors, and extends to multiple testing with a new interpretive scale.

Findings

Empirical Bayes factors are closely related to information criteria.

Bias in log Bayes factors is proportional to the number of parameters.

Approximate Bayes factors from P-values are 10p.

Abstract

Bayes factors for composite hypotheses have difficulty in encoding vague prior knowledge, as improper priors cannot be used and objective priors may be subjectively unreasonable. To address these issues I revisit the posterior Bayes factor, in which the posterior distribution from the data at hand is re-used in the Bayes factor for the same data. I argue that this is biased when calibrated against proper Bayes factors, but propose adjustments to allow interpretation on the same scale. In the important case of a regular normal model, the bias in log scale is half the number of parameters. The resulting empirical Bayes factor is closely related to the widely applicable information criterion. I develop test-based empirical Bayes factors for several standard tests and propose an extension to multiple testing closely related to the optimal discovery procedure. When only a P-value is available, an approximate empirical Bayes factor is 10p. I propose interpreting the strength of Bayes factors on a logarithmic scale with base 3.73, reflecting the sharpest distinction between weaker and stronger belief. This provides an objective framework for interpreting statistical evidence, and realises a Bayesian/frequentist compromise.