Test Martingales, Bayes Factors and $p$-Values

TL;DR

This paper explores the relationships between test martingales, Bayes factors, and p-values, providing methods to control exaggeration in evidence measures and characterizing functions that eliminate such exaggeration.

Contribution

It introduces a framework linking martingales, Bayes factors, and p-values, and characterizes functions that prevent evidence exaggeration in sequential testing.

Findings

Largest martingale value can be used to interpret evidence.

Methods to systematically eliminate evidence exaggeration.

Characterization of functions that remove exaggeration.

Abstract

A nonnegative martingale with initial value equal to one measures evidence against a probabilistic hypothesis. The inverse of its value at some stopping time can be interpreted as a Bayes factor. If we exaggerate the evidence by considering the largest value attained so far by such a martingale, the exaggeration will be limited, and there are systematic ways to eliminate it. The inverse of the exaggerated value at some stopping time can be interpreted as a $p$-value. We give a simple characterization of all increasing functions that eliminate the exaggeration.