Conformal Blindness: A Note on $A$-Cryptic change-points

TL;DR

This paper reveals that conformal test martingales can be blind to certain distribution shifts, called conformal blindness, where p-values remain uniform despite underlying changes, highlighting a fundamental limitation in change-point detection.

Contribution

The paper introduces the concept of conformal blindness and demonstrates the existence of A-cryptic change-points where CTMs fail to detect shifts, especially with ideal conformity measures.

Findings

Conformal blindness can occur even with perfect conformity measures.

A line of change in Gaussian distributions can produce uniform p-values despite distribution shifts.

Simulations confirm the cryptic nature of certain distribution changes.

Abstract

Conformal Test Martingales (CTMs) are a standard method within the Conformal Prediction framework for testing the crucial assumption of data exchangeability by monitoring deviations from uniformity in the p-value sequence. Although exchangeability implies uniform p-values, the converse does not hold. This raises the question of whether a significant break in exchangeability can occur, such that the p-values remain uniform, rendering CTMs blind. We answer this affirmatively, demonstrating the phenomenon of \emph{conformal blindness}. Through explicit construction, for the theoretically ideal ``predictive oracle'' conformity measure (given by the true conditional density), we demonstrate the possibility of an \emph{$A$-cryptic change-point} (where $A$ refers to the conformity measure). Using bivariate Gaussian distributions, we identify a line along which a change in the marginal means does not alter the distribution of the conformity scores, thereby producing perfectly uniform p-values. Simulations confirm that even a massive distribution shift can be perfectly cryptic to the CTM, highlighting a fundamental limitation and emphasising the critical role of the alignment of the conformity measure with potential shifts. By contrasting the predictive oracle with recent results on detection-optimal scores, we emphasise that validity monitoring in safety-critical systems requires careful separation of predictive and diagnostic goals.