Drift Localization using Conformal Predictions

TL;DR

This paper introduces a novel drift localization method using conformal predictions, addressing limitations of existing local testing schemes, and demonstrates its effectiveness on image datasets.

Contribution

Proposes a new conformal prediction-based approach for drift localization, overcoming high-dimensional, low-signal challenges in monitoring distribution changes.

Findings

Outperforms existing local testing methods in high-dimensional settings

Effective on state-of-the-art image datasets

Highlights shortcomings of common drift detection approaches

Abstract

Concept drift -- the change of the distribution over time -- poses significant challenges for learning systems and is of central interest for monitoring. Understanding drift is thus paramount, and drift localization -- determining which samples are affected by the drift -- is essential. While several approaches exist, most rely on local testing schemes, which tend to fail in high-dimensional, low-signal settings. In this work, we consider a fundamentally different approach based on conformal predictions. We discuss and show the shortcomings of common approaches and demonstrate the performance of our approach on state-of-the-art image datasets.