Is this Change the Answer to that Problem? Correlating Descriptions of Bug and Code Changes for Evaluating Patch Correctness

TL;DR

This paper introduces Quatrain, a novel approach that uses natural language processing to assess patch correctness by evaluating the semantic correlation between bug reports and patch descriptions, achieving high accuracy on large datasets.

Contribution

The paper presents Quatrain, a new neural network-based method that correlates bug reports and patch descriptions to evaluate patch correctness, transforming the problem into a question-answering task.

Findings

Achieves an AUC of 0.886 in predicting patch correctness.

Recalls 93% of correct patches while filtering out 62% of incorrect ones.

Demonstrates the importance of input quality on prediction performance.

Abstract

In this work, we propose a novel perspective to the problem of patch correctness assessment: a correct patch implements changes that "answer" to a problem posed by buggy behaviour. Concretely, we turn the patch correctness assessment into a Question Answering problem. To tackle this problem, our intuition is that natural language processing can provide the necessary representations and models for assessing the semantic correlation between a bug (question) and a patch (answer). Specifically, we consider as inputs the bug reports as well as the natural language description of the generated patches. Our approach, Quatrain, first considers state of the art commit message generation models to produce the relevant inputs associated to each generated patch. Then we leverage a neural network architecture to learn the semantic correlation between bug reports and commit messages. Experiments on a large dataset of 9135 patches generated for three bug datasets (Defects4j, Bugs.jar and Bears) show that Quatrain can achieve an AUC of 0.886 on predicting patch correctness, and recalling 93% correct patches while filtering out 62% incorrect patches. Our experimental results further demonstrate the influence of inputs quality on prediction performance. We further perform experiments to highlight that the model indeed learns the relationship between bug reports and code change descriptions for the prediction. Finally, we compare against prior work and discuss the benefits of our approach.