Pitfalls of Unlabeled Disagreement-Based Drift Detection in Streaming Tree Ensembles

TL;DR

This paper examines the limitations of disagreement-based drift detection in streaming ensembles of incremental decision trees, highlighting challenges due to their structural rigidity and limited adaptability.

Contribution

It provides an empirical evaluation of disagreement measures in IDT ensembles, revealing their underperformance and suggesting restructuring approaches for better drift detection.

Findings

Disagreement-based detection works well with neural networks but not with IDTs.

IDTs' structural rigidity limits their ability to reflect concept drift through disagreement.

Restructuring IDTs into non-overlapping rules may improve drift detection.

Abstract

Detecting concept drift in high-speed data streams remains challenging, particularly when models must operate on unlabeled data and avoid false alarms caused by benign shifts. While disagreement-based uncertainty has shown promise in neural networks, its adaptation to ensembles of incremental decision trees (IDTs) remains largely unexplored. We investigate this approach by constructing batch-specific disagreement measures via label flipping in ensemble members and evaluating their effectiveness for drift detection in tabular data streams. Our experiments show that, although this method performs well in ensembles of multi-layer perceptrons (MLPs), it consistently underperforms loss-based detectors when applied to IDTs. We attribute this behavior to the intrinsic rigidity of IDTs: learning primarily through structural expansion, with limited parameter adaptation, restricts model plasticity and prevents disagreement from reliably reflecting learning potential. Recent work on restructuring IDTs using their intrinsic decomposition into non-overlapping rules offers a promising direction for improving adaptability.