Considerations in the Interpretation of Cosmological Anomalies

TL;DR

This paper discusses the challenges of interpreting cosmological anomalies, emphasizing the need for physical insight, careful experimental design, and robust statistical methods to avoid false discoveries.

Contribution

It highlights key considerations for anomaly detection in cosmology, advocating for integrated approaches combining physics, experiment design, and statistics.

Findings

Anomalies often arise from systematics and low signal-to-noise data.

Proper interpretation requires understanding of physical models and experimental biases.

Robust statistical methods are essential to distinguish true anomalies from false positives.

Abstract

Anomalies drive scientific discovery -- they are associated with the cutting edge of the research frontier, and thus typically exploit data in the low signal-to-noise regime. In astronomy, the prevalence of systematics --- both "known unknowns" and "unknown unknowns" --- combined with increasingly large datasets, the widespread use of ad hoc estimators for anomaly detection, and the "look-elsewhere" effect, can lead to spurious false detections. In this informal note, I argue that anomaly detection leading to discoveries of new physics requires a combination of physical understanding, careful experimental design to avoid confirmation bias, and self-consistent statistical methods. These points are illustrated with several concrete examples from cosmology.