A Demonstration of Benchmarking Time Series Management Systems in the Cloud

TL;DR

This paper introduces an interactive benchmark tool for evaluating and comparing the performance of four prominent Time Series Management Systems on advanced analytical operators using real-world datasets.

Contribution

It presents a new benchmark that assesses TSMS performance on complex operators, addressing limitations of existing benchmarks that focus only on simple operators.

Findings

Benchmark supports four TSMS: TimescaleDB, MonetDB, ExtremeDB, Kairos-H2.

Evaluates over 13 advanced analytical operators.

Provides tailored recommendations based on user-selected datasets and operators.

Abstract

Time Series Management Systems (TSMS) are Database Management Systems that have been configured with the primary objective of processing and storing time series data. With the IoT expanding at exponential rates and there becoming increasingly more time series data to process and analyze, several TSMS have been proposed and are used in practice. Each system has its own architecture and storage mechanisms and factors such as the dimensionality of the dataset or the nature of the operators a user wishes to execute can cause differences in system performance. This makes it highly challenging for practitioners to determine the most optimal TSMS for their use case. To remedy this several TSMS benchmarks have been proposed, yet these benchmarks focus primary on simple and supported operators, largely disregarding the advanced analytical operators (ie. Normalization, Clustering, etc) that constitute a large part of the use cases in practice. In this demo, we introduce a new benchmark that enables users to evaluate the performance of four prominent TSMS (TimescaleDB, MonetDB, ExtremeDB, Kairos-H2) in their handling of over 13 advanced analytical operators. In a simple and interactive manner, users can specify the TSMS(s) to compare, the advanced analytical operator(s) to execute, and the dataset(s) to utilize for the comparison. Users can choose from over eight real-world datasets with varying dimensions or upload their own dataset. The tool then provides a report and recommendation of the most optimal TSMS for the parameters chosen.