RePaCA: Leveraging Reasoning Large Language Models for Static Automated Patch Correctness Assessment

TL;DR

RePaCA introduces a novel static patch correctness assessment method leveraging reasoning large language models, significantly improving accuracy, generalization, and explainability in identifying overfitting patches in automated program repair.

Contribution

The paper presents RePaCA, a static APCA technique that uses finetuned reasoning LLMs with Chain of Thought prompting and reinforcement learning, achieving state-of-the-art performance.

Findings

Achieves 83.1% accuracy and 84.8% F1-score on Defects4J-derived test.

Outperforms existing static APCA techniques in accuracy and generalization.

Provides enhanced explainability through reasoning-based assessment.

Abstract

Automated Program Repair (APR) seeks to automatically correct software bugs without requiring human intervention. However, existing tools tend to generate patches that satisfy test cases without fixing the underlying bug, those are known as overfitting patches. To address this issue, Automated Patch Correctness Assessment (APCA) attempts to identify overfitting patches generated by APR tools. It can be solved as a static approach, meaning that no additional information is needed beyond the original and fixed code snippets. Current static techniques often struggle with reliability, flexibility and transparency. To address these issues, we introduce RePaCA, a novel static APCA technique that leverages Large Language Models (LLMs) specialized in thinking tasks. Our model is prompted with both buggy and fixed code snippets and guided to generate a Chain of Thought that analyses code differences, reasons about how the patch addresses the root cause, and ultimately provides a binary classification: correct or overfitting. To enhance these reasoning capabilities for the APCA task specifically, the LLM is finetuned using Reinforcement Learning with the Group Relative Policy Optimization algorithm. When evaluated on a standard Defects4J-derived test, our approach achieves state-of-the-art performance, with 83.1% accuracy and an 84.8% F1-score. Furthermore, our model demonstrates superior generalization capabilities when trained on different datasets, outperforming the leading technique. This reasoning capability also provides enhanced explainability for the patch assessment. These findings underscore the considerable promise of finetuned, reasoning LLMs to advance static APCA by enhancing accuracy, generalization, and explainability.