Doric: Foundations for Statistical Fault Localisation

TL;DR

Doric introduces a formal probabilistic foundation for fault localisation, leading to a new measure called cl that improves accuracy and interpretability over existing spectrum-based heuristics in large-scale software bug detection.

Contribution

The paper presents Doric, a novel formal framework for statistical fault localisation, and derives cl, a lightweight, probabilistic measure that outperforms existing heuristics in fault detection.

Findings

cl finds faults after inspecting 6 lines 41.18% of the time in Defects4J.

cl outperforms all 127 known spectrum-based heuristics.

On Steimann's benchmarks, cl reduces the number of methods to inspect from 9.02 to 5.02.

Abstract

To fix a software bug, you must first find it. As software grows in size and complexity, finding bugs is becoming harder. To solve this problem, measures have been developed to rank lines of code according to their "suspiciousness" wrt being faulty. Engineers can then inspect the code in descending order of suspiciousness until a fault is found. Despite advances, ideal measures --- ones which are at once lightweight, effective, and intuitive --- have not yet been found. We present Doric, a new formal foundation for statistical fault localisation based on classical probability theory. To demonstrate Doric's versatility, we derive cl, a lightweight measure of the likelihood some code caused an error. cl returns probabilities, when spectrum-based heuristics (sbhs) usually return difficult to interpret scores. cl handles fundamental fault scenarios that spectrum-based measures cannot and can also meaningfully identify causes with certainty. We demonstrate its effectiveness in, what is to our knowledge, the largest scale experiment in the fault localisation literature. For Defects4J benchmarks, cl permits a developer to find a fault after inspecting 6 lines of code 41.18% of the time. Furthermore, cl is more accurate at locating faults than all known 127 sbh. In particular, on Steimann's benchmarks one would expect to find a fault by investigating 5.02 methods, as opposed to 9.02 with the best performing sbh.