No Free Lunch But A Cheaper Supper: A General Framework for Streaming Anomaly Detection

TL;DR

This paper introduces SAFARI, a flexible framework for streaming anomaly detection that enables comparison of various algorithms and highlights the absence of a universally best detector across diverse datasets.

Contribution

SAFARI unifies fundamental tasks in streaming anomaly detection, allowing for flexible algorithm implementation and comprehensive performance evaluation.

Findings

No single detector outperforms others across all datasets.

Different algorithms excel in different scenarios.

SAFARI facilitates detailed analysis of algorithm strengths and weaknesses.

Abstract

In recent years, there has been increased research interest in detecting anomalies in temporal streaming data. A variety of algorithms have been developed in the data mining community, which can be divided into two categories (i.e., general and ad hoc). In most cases, general approaches assume the one-size-fits-all solution model where a single anomaly detector can detect all anomalies in any domain. To date, there exists no single general method that has been shown to outperform the others across different anomaly types, use cases and datasets. In this paper, we propose SAFARI, a general framework formulated by abstracting and unifying the fundamental tasks in streaming anomaly detection, which provides a flexible and extensible anomaly detection procedure to overcome the limitations of one-size-fits-all solutions. SAFARI helps to facilitate more elaborate algorithm comparisons by allowing us to isolate the effects of shared and unique characteristics of different algorithms on detection performance. Using SAFARI, we have implemented various anomaly detectors and identified a research gap that motivates us to propose a novel learning strategy in this work. We conducted an extensive evaluation study of 20 detectors that are composed using SAFARI and compared their performances using real-world benchmark datasets with different properties. The results indicate that there is no single superior detector that works well for every case, proving our hypothesis that "there is no free lunch" in the streaming anomaly detection world. Finally, we discuss the benefits and drawbacks of each method in-depth and draw a set of conclusions to guide future users of SAFARI.