Accelerating Automatic Program Repair with Dual Retrieval-Augmented Fine-Tuning and Patch Generation on Large Language Models

TL;DR

This paper introduces SelRepair, a novel approach that combines fine-tuned large language models with a dual retrieval-augmented generation module to improve automated program repair efficiency and accuracy, especially for Java code.

Contribution

The paper presents a new APR method integrating a fine-tuned LLM with a dual RAG module, enhancing retrieval relevance and reducing inference time compared to existing approaches.

Findings

Achieves 26.29% and 17.64% EM on Java datasets.

Reduces inference time by at least 6.42%.

Effectively incorporates semantic and structural code features.

Abstract

Automated Program Repair (APR) is essential for ensuring software reliability and quality while enhancing efficiency and reducing developers' workload. Although rule-based and learning-based APR methods have demonstrated their effectiveness, their performance was constrained by the defect type of repair, the quality of training data, and the size of model parameters. Recently, Large Language Models (LLMs) combined with Retrieval-Augmented-Generation (RAG) have been increasingly adopted in APR tasks. However, current code LLMs and RAG designs neither fully address code repair tasks nor consider code-specific features. To overcome these limitations, we propose SelRepair, a novel APR approach with integration of a fine-tuned LLM with a newly-designed dual RAG module. This approach uses a bug-fix pair dataset for fine-tuning and incorporates semantic and syntactic/structural similarity information through an RAG selection gate. This design ensures relevant information is retrieved efficiently, thereby reducing token length and inference time. Evaluations on Java datasets show SelRepair outperforms other APR methods, achieving 26.29% and 17.64% in terms of exact match (EM) on different datasets while reducing inference time by at least 6.42% with controlled input lengths.