A Novel Algorithm for Optimized Real Time Anomaly Detection in Timeseries

TL;DR

This paper introduces a new real-time anomaly detection algorithm for time series data that effectively handles seasonal and non-seasonal patterns, identifies local and global outliers, and outperforms existing methods in experiments.

Contribution

The paper proposes a novel polynomial curve fitting-based algorithm with a filter for significance, capable of real-time anomaly detection across various data types, improving over traditional statistical models.

Findings

Outperforms existing algorithms on real-time and artificial datasets.

Effectively detects both global and local (contextual) anomalies.

Handles seasonal and non-seasonal data without issues like heteroskedasticity.

Abstract

Observations in data which are significantly different from its neighbouring points but cannot be classified as noise are known as anomalies or outliers. These anomalies are a cause of concern and a timely warning about their presence could be valuable. In this paper, we have evaluated and compared the performance of popular algorithms from domains of Machine Learning and Statistics in detecting anomalies on both offline data as well as real time data. Our aim is to come up with an algorithm which can handle all types of seasonal and non-seasonal data effectively and is fast enough to be of practical utility in real time. It is not only important to detect anomalies at the global but also the ones which are anomalies owing to their local surroundings. Such outliers can be termed as contextual anomalies as they derive their context from the neighbouring observations. Also, we require a methodology to automatically determine the presence of seasonality in the given data. For detecting the seasonality, the proposed algorithm takes up a curve fitting approach rather than model based anomaly detection. The proposed model also introduces a unique filter which assess the relative significance of local outliers and removes the ones deemed as insignificant. Since, the proposed model fits polynomial in buckets of timeseries data, it does not suffer from problems such as heteroskedasticity and breakout as compared to its statistical alternatives such as ARIMA, SARIMA and Winter Holt. Experimental results the proposed algorithm performs better on both real time as well as artificial generated datasets.