The Window Dilemma: Why Concept Drift Detection is Ill-Posed

TL;DR

This paper argues that concept drift detection in data streams is fundamentally ill-posed and often misleading, as perceived drifts are influenced by windowing choices rather than true data changes, questioning the effectiveness of current detectors.

Contribution

It introduces the Window Dilemma, demonstrating that perceived drift depends on windowing and that drift detection is inherently ill-posed, challenging existing methods.

Findings

Traditional batch learning often outperforms drift-aware methods.

Perceived drift is heavily influenced by windowing choices.

Drift detection may not reliably reflect true data changes.

Abstract

Non-stationarity of an underlying data generating process that leads to distributional changes over time is a key characteristic of Data Streams. This phenomenon, commonly referred to as Concept Drift, has been intensively studied, and Concept Drift Detectors have been established as a class of methods for detecting such changes (drifts). For the most part, Drift Detectors compare regions (windows) of the data stream and detect drift if those windows are sufficiently dissimilar. In this work, we introduce the Window Dilemma, an observation that perceived drift is a product of windowing and not necessarily the underlying data generating process. Additionally, we highlight that drift detection is ill-posed, primarily because verification of drift events are implausible in practice. We demonstrate these contributions first by an illustrative example, followed by empirical comparisons of drift detectors against a variety of alternative adaptation strategies. Our main finding is that traditional batch learning techniques often perform better than their drift-aware counterparts further bringing into question the purpose of detectors in Stream Classification.