When Your Model Stops Working: Anytime-Valid Calibration Monitoring

TL;DR

PITMonitor is a novel calibration monitoring method that provides formal error guarantees and detects distributional shifts in probabilistic models over unbounded streams, improving reliability in model deployment.

Contribution

It introduces PITMonitor, an anytime-valid, calibration-specific monitor using a mixture e-process for distributional shift detection with Type I error control.

Findings

Achieves competitive detection rates on benchmark datasets.

Provides formal Type I error guarantees over unbounded streams.

Longer detection delay observed under local drift scenarios.

Abstract

Practitioners monitoring deployed probabilistic models face a fundamental trap: any fixed-sample test applied repeatedly over an unbounded stream will eventually raise a false alarm, even when the model remains perfectly stable. Existing methods typically lack formal error guarantees, conflate alarm time with changepoint location, and monitor indirect signals that do not fully characterize calibration. We present PITMonitor, an anytime-valid calibration-specific monitor that detects distributional shifts in probability integral transforms via a mixture e-process, providing Type I error control over an unbounded monitoring horizon as well as Bayesian changepoint estimation. On river's FriedmanDrift benchmark, PITMonitor achieves detection rates competitive with the strongest baselines across all three scenarios, although detection delay is substantially longer under local drift.