Adaptive Model Pooling for Online Deep Anomaly Detection from a Complex Evolving Data Stream

TL;DR

This paper introduces ARCUS, an adaptive framework for online deep anomaly detection that effectively manages complex and evolving data streams by dynamically pooling models, significantly improving detection accuracy over existing methods.

Contribution

The paper proposes ARCUS, a novel adaptive model pooling framework with concept-driven inference and drift-aware updates for improved online anomaly detection.

Findings

ARCUS outperforms existing methods by up to 22% in accuracy.

It effectively handles high-dimensional, concept-drifted data streams.

Experimental results demonstrate significant accuracy improvements.

Abstract

Online anomaly detection from a data stream is critical for the safety and security of many applications but is facing severe challenges due to complex and evolving data streams from IoT devices and cloud-based infrastructures. Unfortunately, existing approaches fall too short for these challenges; online anomaly detection methods bear the burden of handling the complexity while offline deep anomaly detection methods suffer from the evolving data distribution. This paper presents a framework for online deep anomaly detection, ARCUS, which can be instantiated with any autoencoder-based deep anomaly detection methods. It handles the complex and evolving data streams using an adaptive model pooling approach with two novel techniques: concept-driven inference and drift-aware model pool update; the former detects anomalies with a combination of models most appropriate for the complexity, and the latter adapts the model pool dynamically to fit the evolving data streams. In comprehensive experiments with ten data sets which are both high-dimensional and concept-drifted, ARCUS improved the anomaly detection accuracy of the streaming variants of state-of-the-art autoencoder-based methods and that of the state-of-the-art streaming anomaly detection methods by up to 22% and 37%, respectively.