OrionBench: Benchmarking Time Series Generative Models in the Service of the End-User

TL;DR

OrionBench is a continuously updated benchmarking framework for unsupervised time series anomaly detection models, enabling fair comparison, extensibility, and real-world scenario testing over multiple years.

Contribution

It introduces a novel, end-user-centric benchmarking framework with universal model abstractions, hyperparameter standardization, and ongoing updates for time series anomaly detection.

Findings

Performance varied across 17 releases over four years.

Continuous benchmarking reveals model improvements and stability.

Real-world scenarios highlight the importance of ongoing evaluation.

Abstract

Time series anomaly detection is a vital task in many domains, including patient monitoring in healthcare, forecasting in finance, and predictive maintenance in energy industries. This has led to a proliferation of anomaly detection methods, including deep learning-based methods. Benchmarks are essential for comparing the performances of these models as they emerge, in a fair, rigorous, and reproducible approach. Although several benchmarks for comparing models have been proposed, these usually rely on a one-time execution over a limited set of datasets, with comparisons restricted to a few models. We propose OrionBench: an end-user centric, continuously maintained benchmarking framework for unsupervised time series anomaly detection models. Our framework provides universal abstractions to represent models, hyperparameter standardization, extensibility to add new pipelines and datasets, pipeline verification, and frequent releases with published updates of the benchmark. We demonstrate how to use OrionBench, and the performance of pipelines across 17 releases published over the course of four years. We also walk through two real scenarios we experienced with OrionBench that highlight the importance of continuous benchmarking for unsupervised time series anomaly detection.