Frequentist Evaluation of Intervals Estimated for a Binomial Parameter and for the Ratio of Poisson Means

TL;DR

This paper evaluates various confidence intervals for binomial parameters and Poisson ratios in high energy physics, emphasizing the importance of averaging over observed data for better coverage properties.

Contribution

It introduces an unconditional averaging approach over observed data for interval evaluation, and recommends specific intervals like Clopper-Pearson and Lancaster's mid-P for optimal coverage.

Findings

Lancaster's mid-P intervals provide excellent average coverage.

Clopper-Pearson intervals are recommended for strict conditional coverage.

Unconditional averaging over data offers a better assessment of interval performance.

Abstract

Confidence intervals for a binomial parameter or for the ratio of Poisson means are commonly desired in high energy physics (HEP) applications such as measuring a detection efficiency or branching ratio. Due to the discreteness of the data, in both of these problems the frequentist coverage probability unfortunately depends on the unknown parameter. Trade-offs among desiderata have led to numerous sets of intervals in the statistics literature, while in HEP one typically encounters only the classic intervals of Clopper-Pearson (central intervals with no undercoverage but substantial over-coverage) or a few approximate methods which perform rather poorly. If strict coverage is relaxed, some sort of averaging is needed to compare intervals. In most of the statistics literature, this averaging is over different values of the unknown parameter, which is conceptually problematic from the frequentist point of view in which the unknown parameter is typically fixed. In contrast, we perform an (unconditional) {\it average over observed data} in the ratio-of-Poisson-means problem. If strict conditional coverage is desired, we recommend Clopper-Pearson intervals and intervals from inverting the likelihood ratio test (for central and non-central intervals, respectively). Lancaster's mid-$P$ modification to either provides excellent unconditional average coverage in the ratio-of-Poisson-means problem.