Quilt: Robust Data Segment Selection against Concept Drifts

TL;DR

Quilt is a data-centric framework that improves model accuracy in streaming data with concept drifts by intelligently selecting and discarding data segments, outperforming existing methods in accuracy and efficiency.

Contribution

It introduces Quilt, a novel data segment selection approach that explicitly utilizes drifted data and extends subset selection techniques for better accuracy and efficiency in concept drift scenarios.

Findings

Outperforms state-of-the-art drift adaptation methods

Effectively discards drifted data segments

Maintains high model accuracy with reduced training data

Abstract

Continuous machine learning pipelines are common in industrial settings where models are periodically trained on data streams. Unfortunately, concept drifts may occur in data streams where the joint distribution of the data X and label y, P(X, y), changes over time and possibly degrade model accuracy. Existing concept drift adaptation approaches mostly focus on updating the model to the new data possibly using ensemble techniques of previous models and tend to discard the drifted historical data. However, we contend that explicitly utilizing the drifted data together leads to much better model accuracy and propose Quilt, a data-centric framework for identifying and selecting data segments that maximize model accuracy. To address the potential downside of efficiency, Quilt extends existing data subset selection techniques, which can be used to reduce the training data without compromising model accuracy. These techniques cannot be used as is because they only assume virtual drifts where the posterior probabilities P(y|X) are assumed not to change. In contrast, a key challenge in our setup is to also discard undesirable data segments with concept drifts. Quilt thus discards drifted data segments and selects data segment subsets holistically for accurate and efficient model training. The two operations use gradient-based scores, which have little computation overhead. In our experiments, we show that Quilt outperforms state-of-the-art drift adaptation and data selection baselines on synthetic and real datasets.