Searching for Multi-Fault Programs in Defects4J

TL;DR

This paper identifies and compiles multiple-fault versions within the Defects4J dataset, creating a benchmark to facilitate research on debugging techniques for programs with coexisting faults.

Contribution

The authors systematically find and document multi-fault versions in Defects4J, extending it as a benchmark for multi-fault debugging research.

Findings

Over 95% of studied versions contain multiple faults.

Identified 2 to 24 faults in these versions.

Provides a new platform for multi-fault debugging research.

Abstract

Defects4J has enabled numerous software testing and debugging research work since its introduction. A large part of its contribution, and the resulting popularity, lies in the clear separation and distillation of the root cause of each individual test failure based on careful manual analysis, which in turn allowed researchers to easily study individual faults in isolation. However, in a realistic debugging scenario, multiple faults can coexist and affect test results collectively. Study of automated debugging techniques for these situations, such as failure clustering or fault localisation for multiple faults, would significantly benefit from a reliable benchmark of multiple, coexisting faults. We search for versions of Defects4J subjects that contain multiple faults, by iteratively transplanting fault-revealing test cases across Defects4J versions. Out of 326 studied versions of Defects4J subjects, we report that over 95% (311 versions) actually contain from two to 24 faults. We hope that the extended, multi-fault Defects4J can provide a platform for future research of testing and debugging techniques for multi-fault programs.