DriftGAN: Using historical data for Unsupervised Recurring Drift Detection

TL;DR

DriftGAN is an unsupervised GAN-based approach that detects and identifies recurring concept drifts in data streams, reducing retraining time and outperforming existing methods in various datasets and a real-world astrophysics case.

Contribution

This paper introduces DriftGAN, a novel unsupervised GAN-based method for detecting and recognizing recurring concept drifts in data streams, improving efficiency and accuracy.

Findings

Outperforms state-of-the-art models on multiple datasets

Effectively detects recurring concept drifts in real-world astrophysics data

Reduces data and time needed for model adaptation

Abstract

In real-world applications, input data distributions are rarely static over a period of time, a phenomenon known as concept drift. Such concept drifts degrade the model's prediction performance, and therefore we require methods to overcome these issues. The initial step is to identify concept drifts and have a training method in place to recover the model's performance. Most concept drift detection methods work on detecting concept drifts and signalling the requirement to retrain the model. However, in real-world cases, there could be concept drifts that recur over a period of time. In this paper, we present an unsupervised method based on Generative Adversarial Networks(GAN) to detect concept drifts and identify whether a specific concept drift occurred in the past. Our method reduces the time and data the model requires to get up to speed for recurring drifts. Our key results indicate that our proposed model can outperform the current state-of-the-art models in most datasets. We also test our method on a real-world use case from astrophysics, where we detect the bow shock and magnetopause crossings with better results than the existing methods in the domain.