OLxPBench: Real-time, Semantically Consistent, and Domain-specific are Essential in Benchmarking, Designing, and Implementing HTAP Systems

TL;DR

OLxPBench is a comprehensive HTAP benchmark suite that emphasizes real-time, semantically consistent, and domain-specific workloads, addressing gaps in existing benchmarks for better system evaluation.

Contribution

The paper introduces OLxPBench, a novel HTAP benchmark with hybrid transactions, semantic schema consistency, and combined domain-specific and general workloads.

Findings

Justifies the importance of real-time, semantic, and domain-specific factors in HTAP benchmarking.

Pinpoints bottlenecks in two mainstream distributed HTAP DBMSs.

Provides a publicly available benchmark suite for HTAP system evaluation.

Abstract

As real-time analysis of the new data become increasingly compelling, more organizations deploy Hybrid Transactional/Analytical Processing (HTAP) systems to support real-time queries on data recently generated by online transaction processing. This paper argues that real-time queries, semantically consistent schema, and domain-specific workloads are essential in benchmarking, designing, and implementing HTAP systems. However, most state-of-the-art and state-of-the-practice benchmarks ignore those critical factors. Hence, they are incommensurable and, at worst, misleading in benchmarking, designing, and implementing HTAP systems. This paper presents OLxPBench, a composite HTAP benchmark suite. OLxPBench proposes: (1) the abstraction of a hybrid transaction, performing a real-time query in-between an online transaction, to model widely-observed behavior pattern -- making a quick decision while consulting real-time analysis; (2) a semantically consistent schema to express the relationships between OLTP and OLAP schema; (3) the combination of domain-specific and general benchmarks to characterize diverse application scenarios with varying resource demands. Our evaluations justify the three design decisions of OLxPBench and pinpoint the bottlenecks of two mainstream distributed HTAP DBMSs. International Open Benchmark Council (BenchCouncil) sets up the OLxPBench homepage at https://www.benchcouncil.org/olxpbench/. Its source code is available from https://github.com/BenchCouncil/olxpbench.git.