Evaluating Representation Learning of Code Changes for Predicting Patch Correctness in Program Repair

TL;DR

This paper explores how learned code representations, especially embeddings from neural networks like BERT, can predict patch correctness in program repair, showing promising results comparable to existing dynamic-based methods.

Contribution

It investigates the effectiveness of representation learning approaches for code changes in predicting patch correctness, highlighting the potential of neural embeddings in this task.

Findings

Embeddings from BERT-based models achieved an AUC of about 0.8.

Learned representations are competitive with state-of-the-art dynamic methods.

Representations can complement manually engineered features in patch correctness prediction.

Abstract

A large body of the literature of automated program repair develops approaches where patches are generated to be validated against an oracle (e.g., a test suite). Because such an oracle can be imperfect, the generated patches, although validated by the oracle, may actually be incorrect. While the state of the art explore research directions that require dynamic information or rely on manually-crafted heuristics, we study the benefit of learning code representations to learn deep features that may encode the properties of patch correctness. Our work mainly investigates different representation learning approaches for code changes to derive embeddings that are amenable to similarity computations. We report on findings based on embeddings produced by pre-trained and re-trained neural networks. Experimental results demonstrate the potential of embeddings to empower learning algorithms in reasoning about patch correctness: a machine learning predictor with BERT transformer-based embeddings associated with logistic regression yielded an AUC value of about 0.8 in predicting patch correctness on a deduplicated dataset of 1000 labeled patches. Our study shows that learned representations can lead to reasonable performance when comparing against the state-of-the-art, PATCH-SIM, which relies on dynamic information. These representations may further be complementary to features that were carefully (manually) engineered in the literature.