From Empirical Evaluation to Context-Aware Enhancement: Repairing Regression Errors with LLMs

TL;DR

This paper evaluates the effectiveness of traditional and LLM-based automated program repair techniques on real-world Java regression bugs, introducing a new benchmark and demonstrating the benefits of context-aware enhancements for LLM approaches.

Contribution

It introduces RegMiner4APR, a high-quality Java regression bug benchmark, and empirically shows that context-aware LLM-based APR significantly outperforms non-contextual methods.

Findings

Classical APR tools failed to repair any bugs.

LLM-based APR approaches show promising potential.

Context-aware enhancements improve repair success by 1.8x.

Abstract

[...] Since then, various APR approaches, especially those leveraging the power of large language models (LLMs), have been rapidly developed to fix general software bugs. Unfortunately, the effectiveness of these advanced techniques in the context of regression bugs remains largely unexplored. This gap motivates the need for an empirical study evaluating the effectiveness of modern APR techniques in fixing real-world regression bugs. In this work, we conduct an empirical study of APR techniques on Java regression bugs. To facilitate our study, we introduce RegMiner4APR, a high-quality benchmark of Java regression bugs integrated into a framework designed to facilitate APR research. The current benchmark includes 99 regression bugs collected from 32 widely used real-world Java GitHub repositories. We begin by conducting an in-depth analysis of the benchmark, demonstrating its diversity and quality. Building on this foundation, we empirically evaluate the capabilities of APR to regression bugs by assessing both traditional APR tools and advanced LLM-based APR approaches. Our experimental results show that classical APR tools fail to repair any bugs, while LLM-based APR approaches exhibit promising potential. Motivated by these results, we investigate impact of incorporating bug-inducing change information into LLM-based APR approaches for fixing regression bugs. Our results highlight that this context-aware enhancement significantly improves the performance of LLM-based APR, yielding 1.8x more successful repairs compared to using LLM-based APR without such context.