STEB: In Search of the Best Evaluation Approach for Synthetic Time Series

TL;DR

STEB is a comprehensive benchmark framework designed to objectively evaluate and compare different measures for assessing the quality of synthetic time series data, addressing a key challenge in the field.

Contribution

This paper introduces STEB, the first benchmark for systematic, interpretable, and automated comparison of synthetic time series evaluation measures across diverse datasets and transformations.

Findings

Ranking of 41 evaluation measures established

Upstream embedding choice significantly affects scores

STEB effectively tracks measure reliability and consistency

Abstract

The growing need for synthetic time series, due to data augmentation or privacy regulations, has led to numerous generative models, frameworks, and evaluation measures alike. Objectively comparing these measures on a large scale remains an open challenge. We propose the Synthetic Time series Evaluation Benchmark (STEB) -- the first benchmark framework that enables comprehensive and interpretable automated comparisons of synthetic time series evaluation measures. Using 10 diverse datasets, randomness injection, and 13 configurable data transformations, STEB computes indicators for measure reliability and score consistency. It tracks running time, test errors, and features sequential and parallel modes of operation. In our experiments, we determine a ranking of 41 measures from literature and confirm that the choice of upstream time series embedding heavily impacts the final score.