Comment on "Reproducibility and Replication of Experimental Particle Physics Results"

TL;DR

This paper discusses the importance of replication in high-energy physics, highlighting that many anomalies do not replicate with additional data, which challenges the assumption that current procedures are sufficient for scientific validation.

Contribution

It critically examines the current practices in HEP for ensuring reproducibility and questions the effectiveness of existing safeguards against false positives.

Findings

Anomalies in HEP often fail to replicate with more data.

Current procedures like data blinding and strict thresholds may not prevent false positives.

The paper calls for a reassessment of replication practices in physics.

Abstract

I would like to thank Junk and Lyons (arXiv:2009.06864) for beginning a discussion about replication in high-energy physics (HEP). Junk and Lyons ultimately argue that HEP learned its lessons the hard way through past failures and that other fields could learn from our procedures. They emphasize that experimental collaborations would risk their legacies were they to make a type-1 error in a search for new physics and outline the vigilance taken to avoid one, such as data blinding and a strict $5\sigma$ threshold. The discussion, however, ignores an elephant in the room: there are regularly anomalies in searches for new physics that result in substantial scientific activity but don't replicate with more data.