A Novel Universal Statistic for Computing Upper Limits in Ill-behaved Background

TL;DR

This paper introduces a universal statistical method for setting upper limits in data with unpredictable or non-standard noise, enhancing robustness and efficiency in automated large-scale data analysis.

Contribution

The paper presents a new universal statistic that does not rely on specific distribution assumptions, providing reliable upper limits in complex, non-Gaussian noise environments.

Findings

Effective in non-Gaussian noise conditions

Computationally efficient for large datasets

Provides near-optimal results under expected distributions

Abstract

Analysis of experimental data must sometimes deal with abrupt changes in the distribution of measured values. Setting upper limits on signals usually involves a veto procedure that excludes data not described by an assumed statistical model. We show how to implement statistical estimates of physical quantities (such as upper limits) that are valid without assuming a particular family of statistical distributions, while still providing close to optimal values when the data is from an expected distribution (such as Gaussian or exponential). This new technique can compute statistically sound results in the presence of severe non-Gaussian noise, relaxes assumptions on distribution stationarity and is especially useful in automated analysis of large datasets, where computational speed is important.