Coverage Of Confidence Intervals For Poisson Statistics In Presence Of Systematic Uncertainties

TL;DR

This paper evaluates the coverage properties of confidence intervals for Poisson statistics considering various systematic uncertainties and ordering schemes, extending existing methods to better understand their statistical performance.

Contribution

It extends the Cousins & Highland method to different systematic uncertainties and compares coverage under multiple ordering schemes and ensembles.

Findings

Feldman & Cousins method over-covers with no uncertainties.

Roe & Woodroofe's method under-covers at low signal expectations.

Including uncertainties affects coverage depending on the ensemble considered.

Abstract

In this note we consider coverage of confidence intervals calculated with and without systematic uncertainties. These calculations follow the prescription originally proposed by Cousins & Highland but here extended to account for different shapes, size and types of systematic uncertainties. Also two different ordering schemes are considered: the Feldman & Cousins ordering and its variant where conditioning on the background expectation is applied as proposed by Roe & Woodroofe. Without uncertainties Feldman & Cousins method over-covers as expected because of the discreteness of the Poisson distribution. For Roe & Woodroofe's method we find under-coverage for low signal expectations. When including uncertainties it becomes important to define the ensemble for which the coverage is determined. We consider two different ensembles, where in ensemble A all nuisance parameters are fixed and in ensemble B the nuisance parameters are varied according to their uncertainties. We also discuss the subtleties of the realization of the ensemble with varying systematic uncertainties.