Exploring Generalizable Automated Program Repair with Large Language Models

TL;DR

This paper conducts an extensive empirical evaluation of large language models for automated program repair across multiple programming languages, highlighting their strengths, limitations, and the impact of fault localization on repair accuracy.

Contribution

It provides the first comprehensive analysis of diverse LLMs for APR across different languages and fault localization scenarios, offering insights for developing more generalizable repair techniques.

Findings

Different models excel in different languages.

Combining models improves bug fixing coverage.

Imperfect fault localization significantly reduces repair accuracy.

Abstract

Automated Program Repair (APR) proposes bug fixes to aid developers in maintaining software. The state of the art in this domain focuses on LLMs, leveraging their strong capabilities to comprehend specifications in natural language and to generate program code. However, despite the APR community's research achievements and industry deployments, APR still cannot generalize broadly. In this work, we present an intensive empirical evaluation of LLMs' capabilities in APR. We evaluate a diverse set of 13 recent open and closed models. In particular, we explore language-agnostic repair by utilizing benchmarks for Java, JavaScript, Python, and PHP. Besides the generalization across languages and levels of patch complexity, we also investigate the effects of fault localization (FL). Our key results include: (1) Different LLMs tend to perform best for different languages, which makes it hard to develop cross-platform, single-LLM repair techniques. (2) Combining models by pooling repairs adds value with respect to uniquely fixed bugs, so a committee of expert models should be considered. (3) Under realistic assumptions of imperfect FL, we observe significant drops in accuracy from the usual practice of using perfect FL. Our insights will help develop reliable and generalizable APR techniques and evaluate them in realistic and fair environments.