How good is the Electricity benchmark for evaluating concept drift adaptation

TL;DR

This paper critiques the effectiveness of the Electricity benchmark in evaluating concept drift adaptation, highlighting issues with autocorrelated data that may lead to misleading accuracy improvements.

Contribution

It identifies a problem with using the Electricity benchmark for assessing adaptive classifiers on autocorrelated data, questioning its reliability.

Findings

Random change alarms can artificially inflate accuracy

Autocorrelation in data affects evaluation of adaptation methods

The benchmark may not accurately reflect true adaptation performance

Abstract

In this correspondence, we will point out a problem with testing adaptive classifiers on autocorrelated data. In such a case random change alarms may boost the accuracy figures. Hence, we cannot be sure if the adaptation is working well.