Empirical Evaluation of Large Language Models in Automated Program Repair

TL;DR

This study empirically evaluates large language models for automated program repair across multiple languages and scenarios, revealing that model specialization and prompt design significantly influence repair effectiveness.

Contribution

It provides a comprehensive analysis of modern large-scale LLMs in APR, highlighting the impact of model size, specialization, and prompting strategies on repair performance.

Findings

Model specialization can outperform larger general models.

Repair performance does not increase linearly with model size.

Correct patches often appear early in generation.

Abstract

The increasing prevalence of software bugs has made automated program repair (APR) a key research focus. Large language models (LLMs) offer new opportunities for APR, but existing studies mostly rely on smaller, earlier-generation models and Java benchmarks. The repair capabilities of modern, large-scale LLMs across diverse languages and scenarios remain underexplored. To address this, we conduct a comprehensive empirical study of four open-source LLMs, CodeLlama, LLaMA, StarCoder, and DeepSeek-Coder, spanning 7B to 33B parameters, diverse architectures, and purposes. We evaluate them across two bug scenarios (enterprise-grades and algorithmic), three languages (Java, C/C++, Python), and four prompting strategies, analyzing over 600K generated patches on six benchmarks. Key findings include: (1) model specialization (e.g., CodeLlama) can outperform larger general-purpose models (e.g., LLaMA); (2) repair performance does not scale linearly with model size; (3) correct patches often appear early in generation; and (4) prompts significantly affect results. These insights offer practical guidance for designing effective and efficient LLM-based APR systems.