A Synthetic Benchmark to Explore Limitations of Localized Drift Detections

TL;DR

This paper investigates the challenge of detecting localized concept drift in data streams, introduces a synthetic benchmark for testing drift detection methods, and evaluates the effectiveness of existing techniques in identifying subgroup-specific changes.

Contribution

It presents a synthetic benchmark for localized drift detection and assesses the performance of various methods in identifying subgroup-specific concept drift.

Findings

Common drift detection methods often fail to detect localized drift.

The synthetic benchmark enables systematic evaluation of drift detection techniques.

Some approaches show promise in identifying subgroup-specific concept changes.

Abstract

Concept drift is a common phenomenon in data streams where the statistical properties of the target variable change over time. Traditionally, drift is assumed to occur globally, affecting the entire dataset uniformly. However, this assumption does not always hold true in real-world scenarios where only specific subpopulations within the data may experience drift. This paper explores the concept of localized drift and evaluates the performance of several drift detection techniques in identifying such localized changes. We introduce a synthetic dataset based on the Agrawal generator, where drift is induced in a randomly chosen subgroup. Our experiments demonstrate that commonly adopted drift detection methods may fail to detect drift when it is confined to a small subpopulation. We propose and test various drift detection approaches to quantify their effectiveness in this localized drift scenario. We make the source code for the generation of the synthetic benchmark available at https://github.com/fgiobergia/subgroup-agrawal-drift.