Class Distribution Monitoring for Concept Drift Detection

TL;DR

This paper presents Class Distribution Monitoring (CDM), a novel method for detecting concept drift by monitoring class-conditional distributions, which improves detection accuracy especially when only some classes are affected.

Contribution

The paper introduces CDM, a new nonparametric, class-specific concept drift detection method that outperforms existing approaches in various scenarios.

Findings

CDM detects concept drift more effectively when few classes are affected.

CDM achieves similar detection delays as overall distribution monitoring when all classes drift.

CDM provides better detection than error-based methods when changes are subtle.

Abstract

We introduce Class Distribution Monitoring (CDM), an effective concept-drift detection scheme that monitors the class-conditional distributions of a datastream. In particular, our solution leverages multiple instances of an online and nonparametric change-detection algorithm based on QuantTree. CDM reports a concept drift after detecting a distribution change in any class, thus identifying which classes are affected by the concept drift. This can be precious information for diagnostics and adaptation. Our experiments on synthetic and real-world datastreams show that when the concept drift affects a few classes, CDM outperforms algorithms monitoring the overall data distribution, while achieving similar detection delays when the drift affects all the classes. Moreover, CDM outperforms comparable approaches that monitor the classification error, particularly when the change is not very apparent. Finally, we demonstrate that CDM inherits the properties of the underlying change detector, yielding an effective control over the expected time before a false alarm, or Average Run Length (ARL$_0$).