Drift-Aware Variational Autoencoder-based Anomaly Detection with Two-level Ensembling

TL;DR

This paper introduces VAE++ESDD, a novel drift-aware ensemble approach combining multiple VAEs and concept drift detectors to improve anomaly detection in streaming data with concept drift.

Contribution

It proposes a new ensemble method that integrates incremental learning, multiple VAEs, and drift detectors to enhance anomaly detection in nonstationary environments.

Findings

Significantly outperforms existing methods on real-world datasets.

Effective in environments with low anomaly rates and severe concept drift.

Demonstrates robustness across synthetic and real-world streaming data.

Abstract

In today's digital world, the generation of vast amounts of streaming data in various domains has become ubiquitous. However, many of these data are unlabeled, making it challenging to identify events, particularly anomalies. This task becomes even more formidable in nonstationary environments where model performance can deteriorate over time due to concept drift. To address these challenges, this paper presents a novel method, VAE++ESDD, which employs incremental learning and two-level ensembling: an ensemble of Variational AutoEncoder(VAEs) for anomaly prediction, along with an ensemble of concept drift detectors. Each drift detector utilizes a statistical-based concept drift mechanism. To evaluate the effectiveness of VAE++ESDD, we conduct a comprehensive experimental study using real-world and synthetic datasets characterized by severely or extremely low anomalous rates and various drift characteristics. Our study reveals that the proposed method significantly outperforms both strong baselines and state-of-the-art methods.