Conformance Testing of Relational DBMS Against SQL Specifications

TL;DR

This paper introduces a formal, semantics-based testing method for RDBMSs using Prolog as a reference, effectively identifying bugs and inconsistencies in popular systems by checking adherence to SQL specifications.

Contribution

It presents a novel formal semantics-based testing approach for RDBMS correctness, leveraging Prolog as a reference implementation for differential testing.

Findings

Uncovered 19 bugs in four RDBMSs

Detected 11 inconsistencies related to SQL specification

Proved effectiveness of semantics-based testing approach

Abstract

A Relational Database Management System (RDBMS) is one of the fundamental software that supports a wide range of applications, making it critical to identify bugs within these systems. There has been active research on testing RDBMS, most of which employ crash or use metamorphic relations as the oracle. Although existing approaches can detect bugs in RDBMS, they are far from comprehensively evaluating the RDBMS's correctness (i.e., with respect to the semantics of SQL). In this work, we propose a method to test the semantic conformance of RDBMS i.e., whether its behavior respects the intended semantics of SQL. Specifically, we have formally defined the semantics of SQL and implemented them in Prolog. Then, the Prolog implementation serves as the reference RDBMS, enabling differential testing on existing RDBMS. We applied our approach to four widely-used and thoroughly tested RDBMSs, i.e., MySQL, TiDB, SQLite, and DuckDB. In total, our approach uncovered 19 bugs and 11 inconsistencies, which are all related to violating the SQL specification or missing/unclear specification, thereby demonstrating the effectiveness and applicability of our approach.