Causal Testing: Finding Defects' Root Causes

TL;DR

Causal Testing is a novel approach that uses counterfactual causality to identify root causes of software defects, improving developer accuracy and providing unique insights beyond traditional testing tools.

Contribution

It introduces Causal Testing, a new method leveraging counterfactual causality for root-cause analysis in software debugging, with a prototype implementation.

Findings

Applicable to 71% of real-world defects in Defects4J.

Helps identify root causes in 77% of applicable defects.

Improves defect cause identification accuracy from 80% to 86%.

Abstract

Understanding the root cause of a defect is critical to isolating and repairing buggy behavior. We present Causal Testing, a new method of root-cause analysis that relies on the theory of counterfactual causality to identify a set of executions that likely hold key causal information necessary to understand and repair buggy behavior. Using the Defects4J benchmark, we find that Causal Testing could be applied to 71% of real-world defects, and for 77% of those, it can help developers identify the root cause of the defect. A controlled experiment with 37 developers shows that Causal Testing improves participants' ability to identify the cause of the defect from 80% of the time with standard testing tools to 86% of the time with Causal Testing. The participants report that Causal Testing provides useful information they cannot get using tools such as JUnit. Holmes, our prototype, open-source Eclipse plugin implementation of Causal Testing, is available at http://holmes.cs.umass.edu/.