Integrating Various Software Artifacts for Better LLM-based Bug Localization and Program Repair

TL;DR

This paper introduces DEVLoRe, a novel approach that leverages multiple software artifacts such as issue content, stack traces, and debug info to improve bug localization and program repair using LLMs, outperforming existing methods.

Contribution

DEVLoRe is the first framework to effectively combine diverse software artifacts for LLM-based bug localization and repair, demonstrating significant improvements over state-of-the-art techniques.

Findings

DEVLoRe locates 49.3% of buggy methods in Defects4J v2.0.

DEVLoRe generates 56.0% plausible patches on average.

Incorporating multiple artifacts enhances bug localization and repair performance.

Abstract

LLMs have garnered considerable attention for their potential to streamline Automated Program Repair (APR). LLM-based approaches can either insert the correct code or directly generate patches when provided with buggy methods. However, most of LLM-based APR methods rely on a single type of software information, without fully leveraging different software artifacts. Despite this, many LLM-based approaches do not explore which specific types of information best assist in APR. Addressing this gap is crucial for advancing LLM-based APR techniques. We propose DEVLoRe to use issue content (description and message) and stack error traces to localize buggy methods, then rely on debug information in buggy methods and issue content and stack error to localize buggy lines and generate plausible patches which can pass all unit tests. The results show that while issue content is particularly effective in assisting LLMs with fault localization and program repair, different types of software artifacts complement each other. By incorporating different artifacts, DEVLoRe successfully locates 49.3% and 47.6% of single and non-single buggy methods and generates 56.0% and 14.5% plausible patches for the Defects4J v2.0 dataset, respectively. This outperforms current state-of-the-art APR methods. Furthermore, we re-implemented and evaluated our framework, demonstrating its effectiveness in its effectiveness in resolving 9 unique issues compared to other state-of-the-art frameworks using the same or more advanced models on SWE-bench Lite.We also discussed whether a leading framework for Python code can be directly applied to Java code, or vice versa. The source code and experimental results of this work for replication are available at https://github.com/XYZboom/DEVLoRe.