Suitability of Different Metric Choices for Concept Drift Detection

TL;DR

This paper analyzes various metrics used for unsupervised concept drift detection, comparing their theoretical properties and empirical performance to identify effective approaches for detecting distribution changes over time.

Contribution

The study provides a comprehensive comparison of metrics for drift detection, introduces new metric choices, and evaluates their effectiveness through experiments.

Findings

Certain metrics outperform others in detecting drift accurately

New proposed metrics show improved sensitivity to distribution changes

Empirical results validate theoretical analysis of metric properties

Abstract

The notion of concept drift refers to the phenomenon that the distribution, which is underlying the observed data, changes over time; as a consequence machine learning models may become inaccurate and need adjustment. Many unsupervised approaches for drift detection rely on measuring the discrepancy between the sample distributions of two time windows. This may be done directly, after some preprocessing (feature extraction, embedding into a latent space, etc.), or with respect to inferred features (mean, variance, conditional probabilities etc.). Most drift detection methods can be distinguished in what metric they use, how this metric is estimated, and how the decision threshold is found. In this paper, we analyze structural properties of the drift induced signals in the context of different metrics. We compare different types of estimators and metrics theoretically and empirically and investigate the relevance of the single metric components. In addition, we propose new choices and demonstrate their suitability in several experiments.