LSTM-Based Anomaly Detection: Detection Rules from Extreme Value Theory

TL;DR

This paper introduces an EVT-based anomaly detection method for transportation networks using LSTM prediction errors, demonstrating its superiority over Gaussian-based methods through real-world traffic data analysis.

Contribution

It proposes a novel EVT-based detection rule that does not assume a specific distribution, improving anomaly detection accuracy in transportation data.

Findings

EVT-based detection outperforms Gaussian-based methods

Gaussian assumption is statistically invalid for prediction errors

Proposed method is validated on real-world traffic datasets

Abstract

In this paper, we explore various statistical techniques for anomaly detection in conjunction with the popular Long Short-Term Memory (LSTM) deep learning model for transportation networks. We obtain the prediction errors from an LSTM model, and then apply three statistical models based on (i) the Gaussian distribution, (ii) Extreme Value Theory (EVT), and (iii) the Tukey's method. Using statistical tests and numerical studies, we find strong evidence against the widely employed Gaussian distribution based detection rule on the prediction errors. Next, motivated by fundamental results from Extreme Value Theory, we propose a detection technique that does not assume any parent distribution on the prediction errors. Through numerical experiments conducted on several real-world traffic data sets, we show that the EVT-based detection rule is superior to other detection rules, and is supported by statistical evidence.