PATE: Proximity-Aware Time series anomaly Evaluation

TL;DR

This paper introduces PATE, a new evaluation metric for time series anomaly detection that considers temporal proximity, providing more accurate assessments than traditional metrics.

Contribution

PATE is a novel proximity-aware evaluation metric that improves the assessment of time series anomaly detectors by incorporating temporal relationships and buffer zones.

Findings

PATE outperforms traditional metrics like Point-Adjusted F1 Score.

Experiments on synthetic and real datasets demonstrate PATE's superior evaluation accuracy.

State-of-the-art detectors are more fairly compared using PATE.

Abstract

Evaluating anomaly detection algorithms in time series data is critical as inaccuracies can lead to flawed decision-making in various domains where real-time analytics and data-driven strategies are essential. Traditional performance metrics assume iid data and fail to capture the complex temporal dynamics and specific characteristics of time series anomalies, such as early and delayed detections. We introduce Proximity-Aware Time series anomaly Evaluation (PATE), a novel evaluation metric that incorporates the temporal relationship between prediction and anomaly intervals. PATE uses proximity-based weighting considering buffer zones around anomaly intervals, enabling a more detailed and informed assessment of a detection. Using these weights, PATE computes a weighted version of the area under the Precision and Recall curve. Our experiments with synthetic and real-world datasets show the superiority of PATE in providing more sensible and accurate evaluations than other evaluation metrics. We also tested several state-of-the-art anomaly detectors across various benchmark datasets using the PATE evaluation scheme. The results show that a common metric like Point-Adjusted F1 Score fails to characterize the detection performances well, and that PATE is able to provide a more fair model comparison. By introducing PATE, we redefine the understanding of model efficacy that steers future studies toward developing more effective and accurate detection models.