DiffSpec: Differential Testing with LLMs using Natural Language Specifications and Code Artifacts

TL;DR

DiffSpec leverages large language models to generate targeted differential tests from natural language specifications and code artifacts, effectively uncovering multiple bugs in complex software systems.

Contribution

The paper introduces DiffSpec, a novel framework that uses prompt chaining with LLMs to generate differential tests from natural language specs and code artifacts, revealing previously unknown bugs.

Findings

Generated 1901 tests for eBPF, uncovering four bugs.

Produced 299 tests for Wasm validators, identifying two bugs.

Demonstrated effectiveness on real-world systems like eBPF and Wasm validators.

Abstract

Differential testing can be an effective way to find bugs in software systems with multiple implementations that conform to the same specification, like compilers, network protocol parsers, or language runtimes. Specifications for such systems are often standardized in natural language documents, like Instruction Set Architecture (ISA) specifications or IETF RFC's. Large Language Models (LLMs) have demonstrated potential in both generating tests and handling large volumes of natural language text, making them well-suited for analyzing artifacts like specification documents, bug reports, and code implementations. In this work, we leverage natural language and code artifacts to guide LLMs to generate targeted tests that highlight meaningful behavioral differences between implementations, including those corresponding to bugs. We introduce DiffSpec, a framework for generating differential tests with LLMs using prompt chaining. We demonstrate DiffSpec's efficacy on two different (extensively tested) systems, eBPF runtimes and Wasm validators. Using DiffSpec, we generated 1901 differentiating tests, uncovering at least four distinct and confirmed bugs in eBPF, including a kernel memory leak, inconsistent behavior in jump instructions, undefined behavior when using the stack pointer, and tests with infinite loops that hang the verifier in ebpf-for-windows. We also found 299 differentiating tests in Wasm validators pointing to two confirmed and fixed bugs.