Another Fit Bites the Dust: Conformal Prediction as a Calibration Standard for Machine Learning in High-Energy Physics

TL;DR

This paper demonstrates that conformal prediction provides a universal, calibration layer for machine learning models in high-energy physics, offering valid uncertainty estimates without retraining across various tasks.

Contribution

It introduces conformal prediction as a standard calibration method for diverse ML models in collider physics, ensuring honest uncertainty quantification and error control.

Findings

Conformal prediction applies across regression, classification, anomaly detection, and generative modeling.

It provides statistically valid prediction sets and p-values with finite-sample guarantees.

It does not improve raw model accuracy but enhances uncertainty reliability.

Abstract

Machine-learning techniques are essential in modern collider research, yet their probabilistic outputs often lack calibrated uncertainty estimates and finite-sample guarantees, limiting their direct use in statistical inference and decision-making. Conformal prediction (CP) provides a simple, distribution-free framework for calibrating arbitrary predictive models without retraining, yielding rigorous uncertainty quantification with finite-sample coverage guarantees under minimal exchangeability assumptions, without reliance on asymptotics, limit theorems, or Gaussian approximations. In this work, we investigate CP as a unifying calibration layer for machine-learning applications in high-energy physics. Using publicly available collider datasets and a diverse set of models, we show that a single conformal formalism can be applied across regression, binary and multi-class classification, anomaly detection, and generative modelling, converting raw model outputs into statistically valid prediction sets, typicality regions, and p-values with controlled false-positive rates. While conformal prediction does not improve raw model performance, it enforces honest uncertainty quantification and transparent error control. We argue that conformal calibration should be adopted as a standard component of machine-learning pipelines in collider physics, enabling reliable interpretation, robust comparisons, and principled statistical decisions in experimental and phenomenological analyses.