ResBench: A Comprehensive Framework for Evaluating Database Resilience

TL;DR

ResBench is a comprehensive, automated framework for evaluating database resilience under adverse conditions, offering standardized testing, multi-dimensional metrics, and user-friendly visualization to assess performance during adversity.

Contribution

This paper introduces ResBench, a novel benchmark framework that systematically evaluates database resilience across multiple dimensions with automation and visualization.

Findings

ResBench effectively simulates adverse events during transaction processing.

It evaluates resilience across eight key dimensions.

Results are presented via an intuitive graphical interface.

Abstract

Existing database benchmarks primarily focus on performance under ideal running environments. However, in real-world scenarios, databases probably face numerous adverse events. Quantifying the ability to cope with these events from a comprehensive perspective remains an open problem. We provide the definition of database resilience to describe its performance when facing adversity and propose ResBench, a benchmark for evaluating database resilience. This framework achieves automation, standardization, and visualization of the testing process through clear hierarchical decoupling. ResBench simulates adverse events and injects them during normal transaction processing, utilizing a module to gather multiple metrics for the evaluation model. We assess database resilience across eight dimensions: throughput, latency, stability, resistance, recovery, disturbance period, adaptation capability and metric deviation. All the results are presented to users via a user-friendly graphical interface. We demonstrate the execution process and result interpretation of ResBench using two types of adversity datasets.