Dinkel: State-Aware and Granular Framework for Validating Graph Databases

TL;DR

Dinkel is a comprehensive testing framework for graph database systems that generates complex queries and uses query transformations to validate correctness, uncovering numerous bugs and improving testing coverage.

Contribution

It introduces a novel approach combining state-aware query generation and fine-grained query transformations for thorough GDBMS testing.

Findings

Found 127 bugs across three GDBMSs

Covered over 70% more code than existing methods

Identified and confirmed 113 bugs, with 84 fixed

Abstract

Graph database management systems (GDBMSs) have been powering many data-driven applications. To ensure GDBMS reliability, several testing approaches have been proposed. However, they all suffer from two key limitations: (1) insufficient support for generating complex and valid queries to exercise deep GDBMS code, and (2) lack of general oracles to validate the execution correctness of arbitrary queries. In this paper, we propose a novel and practical approach Dinkel, to thoroughly test GDBMSs. Our approach consists of two core techniques. First, to generate complex and valid queries, we model two kinds of graph state, query context and graph schema, to describe the Cypher variables and the manipulated graph labels and properties. We generate queries clause-by-clause, and modify the graph states on the fly to ensure each clause references the correct state information. Second, to generally validate query results, we introduce two fine-grained query transformations: clause-level and expression-level transformations. These transformations can operate on arbitrary queries while preserving their semantics. Dinkel validates GDBMSs by checking whether the transformed query produces the same results as the original. We evaluated Dinkel on three well-known GDBMSs. In total, we found 127 bugs, among which 113 were confirmed, 84 were fixed, and 33 were logic bugs. Compared to existing approaches, Dinkel can cover over 70% more code and find substantially more bugs within a 48-hour testing campaign. We expect Dinkel's powerful bug detection to lay a practical foundation for GDBMS testing.