Real-Time Anomaly Detection for Advanced Manufacturing: Improving on Twitter's State of the Art

TL;DR

This paper introduces a new, statistically principled algorithm for real-time anomaly detection in streaming time series data, optimized for manufacturing and web applications, outperforming Twitter's existing state-of-the-art method.

Contribution

It adapts the generalized extreme studentised deviate test for streaming data using recursive updates and a priority queue to reduce memory, enhancing real-time anomaly detection performance.

Findings

Outperforms Twitter's AnomalyDetection software in real-time scenarios

Efficient recursive update mechanism reduces computational load

Validated on both unlabelled Twitter data and labelled manufacturing data

Abstract

The detection of anomalies in real time is paramount to maintain performance and efficiency across a wide range of applications including web services and smart manufacturing. This paper presents a novel algorithm to detect anomalies in streaming time series data via statistical learning. We adapt the generalised extreme studentised deviate test [1] to streaming data by using a sliding window approach. This is made computationally feasible by recursive updates of the Grubbs test statistic [2]. Moreover, a priority queue [3] is employed to reduce memory requirements, where subsets of the required data streaming window are maintained in the algorithm rather than the full list. Our method is statistically principled. It is suitable for streaming data and it outperforms the AnomalyDetection software package, recently released by Twitter Inc. (Twitter) [4] and used by multiple teams at Twitter as their state of the art on a daily basis [5]. The methodology is demonstrated using an example of unlabelled data from the Twitter AnomalyDetection GitHub repository and using a real manufacturing example with labelled anomalies.