Exploring and Lifting the Robustness of LLM-powered Automated Program Repair with Metamorphic Testing

TL;DR

This paper introduces MT-LAPR, a metamorphic testing framework for Large Language Model-powered Automated Program Repair, revealing significant robustness issues and proposing a training-based solution to improve repair stability.

Contribution

The paper develops a novel metamorphic testing framework for LAPR, identifies robustness vulnerabilities, and proposes a training method to enhance LAPR robustness using generated test cases.

Findings

34.4% - 48.5% of test cases expose LAPR instability

Robustness correlates positively with code readability

Proposed training approach improves robustness by up to 49.32%

Abstract

In recent years, Large language model-powered Automated Program Repair (LAPR) techniques have achieved state-of-the-art bug-fixing performance and have been pervasively applied and studied in both industry and academia. Nonetheless, LLMs were proved to be highly sensitive to input prompts, with slight differences in the expressions of semantically equivalent programs potentially causing repair failures. Therefore, it is crucial to conduct robustness testing on LAPR techniques before their practical deployment. However, related research is scarce. To this end, we propose MT-LAPR, a Metamorphic Testing framework exclusively for LAPR techniques, which summarizes nine widely-recognized Metamorphic Relations (MRs) by developers across three perturbation levels: token, statement, and block. Afterward, our proposed MRs are applied to buggy codes to generate test cases, which are semantically equivalent yet to affect the inference of LAPR. Experiments are carried out on two extensively examined bug-fixing datasets, i.e., Defect4J and QuixBugs, and four bug-fixing abled LLMs released recently, demonstrating that 34.4% - 48.5% of the test cases expose the instability of LAPR techniques on average, showing the effectiveness of MT-LAPR and uncovering a positive correlation between code readability and the robustness of LAPR techniques. Inspired by the above findings, this paper uses the test cases generated by MT-LAPR as samples to train a CodeT5-based code editing model aiming at improving code readability and then embeds it into the LAPR workflow as a data preprocessing step. Extensive experiments demonstrate that this approach significantly enhances the robustness of LAPR by 49.32% at most.