On methods for correcting for the look-elsewhere effect in searches for new physics

TL;DR

This paper compares statistical methods for correcting the look-elsewhere effect in physics searches, highlighting their performance differences with small and large samples, and providing practical guidelines for method selection.

Contribution

It evaluates two recent methods for multiple hypothesis testing correction in physics, revealing their strengths and limitations across different sample sizes.

Findings

The Pilla et al. method inflates false detection rates with small samples.

Gross and Vitells' method performs consistently across sample sizes.

Counter-intuitive scenarios where corrections are more conservative than computationally efficient methods.

Abstract

The search for new significant peaks over a energy spectrum often involves a statistical multiple hypothesis testing problem. Separate tests of hypothesis are conducted at different locations producing an ensemble of local p-values, the smallest of which is reported as evidence for the new resonance. Unfortunately, controlling the false detection rate (type I error rate) of such procedures may lead to excessively stringent acceptance criteria. In the recent physics literature, two promising statistical tools have been proposed to overcome these limitations. In 2005, a method to "find needles in haystacks" was introduced by Pilla et al. [1], and a second method was later proposed by Gross and Vitells [2] in the context of the "look elsewhere effect" and trial factors. We show that, for relatively small sample sizes, the former leads to an artificial inflation of statistical power that stems from an increase in the false detection rate, whereas the two methods exhibit similar performance for large sample sizes. We apply the methods to realistic simulations of the Fermi Large Area Telescope data, in particular the search for dark matter annihilation lines. Further, we discuss the counter-intutive scenario where the look-elsewhere corrections are more conservative than much more computationally efficient corrections for multiple hypothesis testing. Finally, we provide general guidelines for navigating the tradeoffs between statistical and computational efficiency when selecting a statistical procedure for signal detection.