Look everywhere effects in anomaly detection

TL;DR

This paper investigates the look elsewhere effect in machine learning-based anomaly detection, highlighting calibration challenges and proposing k-folding as a balanced solution, supported by numerical and collider physics studies.

Contribution

It provides a detailed analysis of calibration issues in anomaly detection and evaluates methods like data splitting and k-folding to optimize sensitivity and statistical validity.

Findings

Training and testing on the same data leads to miscalibrated p-values.

Calibrating p-values improves sensitivity but reduces calibration.

K-folding offers a good balance between calibration and sensitivity.

Abstract

Machine learning-based anomaly detection methods are able to search high-dimensional spaces for hints of new physics with much less theory bias than traditional searches. However, by searching in many directions all at once, the statistical power of these search strategies is diluted by a variant of the look elsewhere effect. We examine this challenge in detail, focusing on weakly supervised methods. We find that training and testing on the same data results in badly miscalibrated $p$-values due to the anomaly detector searching everywhere in the data and overfitting on statistical fluctuations. However, if these $p$-values can be calibrated, they may offer the best sensitivity to anomalies, since this approach uses all of the data. Conversely, training on half of the data and testing on the other half results in perfectly calibrated $p$-values, but at the cost of reduced sensitivity to anomalies. Similarly, regularization methods such as early stopping can help with $p$-value calibration but also possibly at the expense of sensitivity. Finally, we find that k-folding strikes an effective balance between calibration and sensitivity. Our findings are supported by numerical studies with Gaussian random variables as well as from collider physics using the LHC Olympics benchmark anomaly detection dataset.