Adaptive Anomaly Detection in the Presence of Concept Drift: Extended Report

TL;DR

This paper presents AnDri, a novel framework for anomaly detection that effectively handles concept drift in time series by dynamically modeling normal patterns and introducing a new clustering method, improving detection accuracy.

Contribution

The paper introduces AnDri, a framework that combines dynamic normal modeling with a new clustering technique to better detect anomalies amidst concept drift.

Findings

AnDri outperforms existing methods on real datasets.

The Adjacent Hierarchical Clustering improves detection of short-lived patterns.

Dynamic normal models adapt effectively to concept drift.

Abstract

The presence of concept drift poses challenges for anomaly detection in time series. While anomalies are caused by undesirable changes in the data, differentiating abnormal changes from varying normal behaviours is difficult due to differing frequencies of occurrence, varying time intervals when normal patterns occur, and identifying similarity thresholds to separate the boundary between normal vs. abnormal sequences. Differentiating between concept drift and anomalies is critical for accurate analysis as studies have shown that the compounding effects of error propagation in downstream tasks lead to lower detection accuracy and increased overhead due to unnecessary model updates. Unfortunately, existing work has largely explored anomaly detection and concept drift detection in isolation. We introduce AnDri, a framework for Anomaly detection in the presence of Drift. AnDri introduces the notion of a dynamic normal model where normal patterns are activated, deactivated or newly added, providing flexibility to adapt to concept drift and anomalies over time. We introduce a new clustering method, Adjacent Hierarchical Clustering (AHC), for learning normal patterns that respect their temporal locality; critical for detecting short-lived, but recurring patterns that are overlooked by existing methods. Our evaluation shows AnDri outperforms existing baselines using real datasets with varying types, proportions, and distributions of concept drift and anomalies.