Performance of Test Supermartingale Confidence Intervals for the Success Probability of Bernoulli Trials

TL;DR

This paper evaluates the effectiveness of test supermartingales in constructing confidence intervals and p-values for Bernoulli trial success probabilities, comparing their performance with traditional bounds and exact methods.

Contribution

It introduces a comparison framework for test supermartingales against classical bounds in Bernoulli trials, highlighting their adaptability and performance costs.

Findings

Test supermartingales provide flexible confidence intervals and p-values.

They can outperform traditional bounds in certain scenarios.

Using test supermartingales incurs a quantifiable cost in efficiency.

Abstract

Given a composite null hypothesis H, test supermartingales are non-negative supermartingales with respect to H with initial value 1. Large values of test supermartingales provide evidence against H. As a result, test supermartingales are an effective tool for rejecting H, particularly when the p-values obtained are very small and serve as certificates against the null hypothesis. Examples include the rejection of local realism as an explanation of Bell test experiments in the foundations of physics and the certification of entanglement in quantum information science. Test supermartingales have the advantage of being adaptable during an experiment and allowing for arbitrary stopping rules. By inversion of acceptance regions, they can also be used to determine confidence sets. We use an example to compare the performance of test supermartingales for computing p-values and confidence intervals to Chernoff-Hoeffding bounds and the "exact" p-value. The example is the problem of inferring the probability of success in a sequence of Bernoulli trials. There is a cost in using a technique that has no restriction on stopping rules, and for a particular test supermartingale, our study quantifies this cost.