On Computing Upper Limits to Source Intensities

TL;DR

This paper formalizes the concept of upper limits in astrophysics detection problems using statistical power and error rates, clarifying their distinction from confidence bounds and providing a universal computation method.

Contribution

It introduces a self-consistent statistical framework for computing upper limits based on TypeII error and detection thresholds, differentiating them from confidence intervals.

Findings

Defines upper limits as detection procedure characteristics.

Provides a universal recipe for computing upper limits.

Clarifies the distinction between upper limits and confidence bounds.

Abstract

A common problem in astrophysics is determining how bright a source could be and still not be detected. Despite the simplicity with which the problem can be stated, the solution involves complex statistical issues that require careful analysis. In contrast to the confidence bound, this concept has never been formally analyzed, leading to a great variety of often ad hoc solutions. Here we formulate and describe the problem in a self-consistent manner. Detection significance is usually defined by the acceptable proportion of false positives (the TypeI error), and we invoke the complementary concept of false negatives (the TypeII error), based on the statistical power of a test, to compute an upper limit to the detectable source intensity. To determine the minimum intensity that a source must have for it to be detected, we first define a detection threshold, and then compute the probabilities of detecting sources of various intensities at the given threshold. The intensity that corresponds to the specified TypeII error probability defines that minimum intensity, and is identified as the upper limit. Thus, an upper limit is a characteristic of the detection procedure rather than the strength of any particular source and should not be confused with confidence intervals or other estimates of source intensity. This is particularly important given the large number of catalogs that are being generated from increasingly sensitive surveys. We discuss the differences between these upper limits and confidence bounds. Both measures are useful quantities that should be reported in order to extract the most science from catalogs, though they answer different statistical questions: an upper bound describes an inference range on the source intensity, while an upper limit calibrates the detection process. We provide a recipe for computing upper limits that applies to all detection algorithms.