Towards Unbiased Evaluation of Time-series Anomaly Detector

TL;DR

This paper highlights biases in current time-series anomaly detection evaluation methods and proposes a new balanced adjustment protocol to ensure fairer and more accurate performance assessment.

Contribution

It introduces a novel Balanced point adjustment method that overcomes biases of heuristic adjustments, providing fairer evaluation of TSAD algorithms.

Findings

Existing point adjustments bias towards true positives

The proposed Balanced adjustment offers fairer performance metrics

Guarantees of fairness are established through axiomatic definitions

Abstract

Time series anomaly detection (TSAD) is an evolving area of research motivated by its critical applications, such as detecting seismic activity, sensor failures in industrial plants, predicting crashes in the stock market, and so on. Across domains, anomalies occur significantly less frequently than normal data, making the F1-score the most commonly adopted metric for anomaly detection. However, in the case of time series, it is not straightforward to use standard F1-score because of the dissociation between `time points' and `time events'. To accommodate this, anomaly predictions are adjusted, called as point adjustment (PA), before the $F_1$-score evaluation. However, these adjustments are heuristics-based, and biased towards true positive detection, resulting in over-estimated detector performance. In this work, we propose an alternative adjustment protocol called ``Balanced point adjustment'' (BA). It addresses the limitations of existing point adjustment methods and provides guarantees of fairness backed by axiomatic definitions of TSAD evaluation.