Stream-based Active Learning with Verification Latency in Non-stationary Environments

TL;DR

This paper addresses the challenge of active learning in data streams with concept drift and verification latency, proposing a new utility estimator and dynamic budget strategy that improve model adaptation and performance.

Contribution

It introduces PRopagate, a latency-independent utility estimator that predicts labels, and a drift-dependent dynamic budget strategy for better active learning in non-stationary environments.

Findings

PRopagate outperforms existing methods in various scenarios.

Dynamic budget allocation enhances active learning effectiveness.

The approach maintains high accuracy despite verification delays.

Abstract

Data stream classification is an important problem in the field of machine learning. Due to the non-stationary nature of the data where the underlying distribution changes over time (concept drift), the model needs to continuously adapt to new data statistics. Stream-based Active Learning (AL) approaches address this problem by interactively querying a human expert to provide new data labels for the most recent samples, within a limited budget. Existing AL strategies assume that labels are immediately available, while in a real-world scenario the expert requires time to provide a queried label (verification latency), and by the time the requested labels arrive they may not be relevant anymore. In this article, we investigate the influence of finite, time-variable, and unknown verification delay, in the presence of concept drift on AL approaches. We propose PRopagate (PR), a latency independent utility estimator which also predicts the requested, but not yet known, labels. Furthermore, we propose a drift-dependent dynamic budget strategy, which uses a variable distribution of the labelling budget over time, after a detected drift. Thorough experimental evaluation, with both synthetic and real-world non-stationary datasets, and different settings of verification latency and budget are conducted and analyzed. We empirically show that the proposed method consistently outperforms the state-of-the-art. Additionally, we demonstrate that with variable budget allocation in time, it is possible to boost the performance of AL strategies, without increasing the overall labeling budget.