Testing For Distribution Shifts with Conditional Conformal Test Martingales

TL;DR

This paper introduces a new sequential testing method for detecting distribution shifts that avoids contamination issues present in existing conformal test martingales, offering faster detection and reliable error control.

Contribution

It presents a robust martingale construction that compares new samples to a fixed reference, improving detection speed and maintaining error control under distribution shifts.

Findings

Detects distribution shifts faster than standard CTMs

Provides anytime-valid type-I error control

Guarantees asymptotic power and bounded detection delay

Abstract

We propose a sequential test for detecting arbitrary distribution shifts that allows conformal test martingales (CTMs) to work under a fixed, reference-conditional setting. Existing CTM detectors construct test martingales by continually growing a reference set with each incoming sample, using it to assess how atypical the new sample is relative to past observations. While this design yields anytime-valid type-I error control, it suffers from test-time contamination: after a change, post-shift observations enter the reference set and dilute the evidence for distribution shift, increasing detection delay and reducing power. In contrast, our method avoids contamination by design by comparing each new sample to a fixed null reference dataset. Our main technical contribution is a robust martingale construction that remains valid conditional on the null reference data, achieved by explicitly accounting for the estimation error in the reference distribution induced by the finite reference set. This yields anytime-valid type-I error control together with guarantees of asymptotic power one and bounded expected detection delay. Empirically, our method detects shifts faster than standard CTMs, providing a powerful and reliable distribution-shift detector.