Unsupervised Concept Drift Detection from Deep Learning Representations in Real-time

TL;DR

This paper introduces DriftLens, an unsupervised, real-time framework for detecting and characterizing concept drift in deep learning models handling unstructured data, outperforming previous methods in accuracy and speed.

Contribution

The paper presents DriftLens, a novel unsupervised approach leveraging deep learning representations for efficient, accurate, and explainable concept drift detection in real-time.

Findings

Outperforms previous methods in 15 out of 17 use cases

Runs at least 5 times faster than existing approaches

Produces drift curves with high correlation (≥0.85) to actual drift

Abstract

Concept drift is the phenomenon in which the underlying data distributions and statistical properties of a target domain change over time, leading to a degradation in model performance. Consequently, production models require continuous drift detection monitoring. Most drift detection methods to date are supervised, relying on ground-truth labels. However, they are inapplicable in many real-world scenarios, as true labels are often unavailable. Although recent efforts have proposed unsupervised drift detectors, many lack the accuracy required for reliable detection or are too computationally intensive for real-time use in high-dimensional, large-scale production environments. Moreover, they often fail to characterize or explain drift effectively. To address these limitations, we propose \textsc{DriftLens}, an unsupervised framework for real-time concept drift detection and characterization. Designed for deep learning classifiers handling unstructured data, \textsc{DriftLens} leverages distribution distances in deep learning representations to enable efficient and accurate detection. Additionally, it characterizes drift by analyzing and explaining its impact on each label. Our evaluation across classifiers and data-types demonstrates that \textsc{DriftLens} (i) outperforms previous methods in detecting drift in 15/17 use cases; (ii) runs at least 5 times faster; (iii) produces drift curves that align closely with actual drift (correlation $\geq\!0.85$); (iv) effectively identifies representative drift samples as explanations.