Bias, variance, and confidence intervals for efficiency estimators in particle physics experiments

TL;DR

This paper analyzes the bias, variance, and confidence intervals of efficiency estimators in particle physics experiments, considering various practical data analysis scenarios and proposing generalized Wilson intervals for improved confidence interval estimation.

Contribution

It introduces generalized Wilson intervals tailored for different complex data scenarios in particle physics, extending traditional methods with closed-form solutions.

Findings

Efficiency estimators are unbiased except when samples are weighted.

Standard Wilson interval works well for Poisson-distributed trials.

Generalized Wilson intervals provide accurate coverage in complex cases.

Abstract

We compute bias, variance, and approximate confidence intervals for the efficiency of a random selection process under various special conditions that occur in practical data analysis. We consider the following cases: a) the number of trials is not constant but drawn from a Poisson distribution, b) the samples are weighted, c) the numbers of successes and failures have a variance which exceeds that of a Poisson process, which is the case, for example, when these numbers are obtained from a fit to mixture of signal and background events. Generalized Wilson intervals based on these variances are computed, and their coverage probability is studied. The efficiency estimators are unbiased in all considered cases, except when the samples are weighted. The standard Wilson interval is also suitable for case a). For most of the other cases, generalized Wilson intervals can be computed with closed-form expressions.