Testing DBMS Performance with Mutations

TL;DR

This paper introduces AMOEBA, a system that automatically detects performance bugs in database management systems by comparing semantically equivalent query pairs and analyzing their response times, uncovering previously unknown issues.

Contribution

AMOEBA employs novel mutation rules and feedback mechanisms to effectively identify performance bugs in DBMSs, demonstrating its success on PostgreSQL and CockroachDB.

Findings

Discovered 20 new performance bugs in PostgreSQL and CockroachDB.

Developers confirmed 14 of these bugs, with 4 already fixed.

AMOEBA improves detection efficiency through feedback mechanisms.

Abstract

Because database systems are the critical component of modern data-intensive applications, it is important to ensure that they operate correctly. To this end, developers extensively test these systems to eliminate bugs that negatively affect functionality. In addition to functional bugs, however, there is another important class of bugs: performance bugs. These bugs negatively affect the response time of a database system and can therefore affect the overall performance of the system. Despite their impact on end-user experience, performance bugs have received considerably less attention than functional bugs. In this paper, we present AMOEBA, a system for automatically detecting performance bugs in database systems. The core idea behind AMOEBA is to construct query pairs that are semantically equivalent to each other and then compare their response time on the same database system. If the queries exhibit a significant difference in their runtime performance, then the root cause is likely a performance bug in the system. We propose a novel set of structure and predicate mutation rules for constructing query pairs that are likely to uncover performance bugs. We introduce feedback mechanisms for improving the efficacy and computational efficiency of the tool. We evaluate AMOEBA on two widely-used DBMSs, namely PostgreSQL and CockroachDB. AMOEBA has discovered 20 previously-unknown performance bugs, among which developers have already confirmed 14 and fixed 4.