Improving LLM-Based Fault Localization with External Memory and Project Context

TL;DR

MemFL enhances LLM-based fault localization by integrating project-specific external memory, significantly improving bug detection accuracy, efficiency, and cost-effectiveness, especially for complex software projects.

Contribution

This paper introduces MemFL, a novel method that incorporates static and dynamic project knowledge into LLMs for more effective fault localization.

Findings

MemFL localized 12.7% more bugs than existing methods on Defects4J.

MemFL reduced debugging time to 17.4 seconds per bug.

MemFL achieved cost savings of 67% per bug.

Abstract

Fault localization, the process of identifying the software components responsible for failures, is essential but often time-consuming. Recent advances in Large Language Models (LLMs) have enabled fault localization without extensive defect datasets or model fine-tuning. However, existing LLM-based methods rely only on general LLM capabilities and lack integration of project-specific knowledge, resulting in limited effectiveness, especially for complex software. We introduce MemFL, a novel approach that enhances LLM-based fault localization by integrating project-specific knowledge via external memory. This memory includes static summaries of the project and dynamic, iterative debugging insights gathered from previous attempts. By leveraging external memory, MemFL simplifies debugging into three streamlined steps, significantly improving efficiency and accuracy. Iterative refinement through dynamic memory further enhances reasoning quality over time. Evaluated on the Defects4J benchmark, MemFL using GPT-4o-mini localized 12.7% more bugs than current LLM-based methods, achieving this improvement with just 21% of the execution time (17.4 seconds per bug) and 33% of the API cost (0.0033 dollars per bug). On complex projects, MemFL's advantage increased to 27.6%. Additionally, MemFL with GPT-4.1-mini outperformed existing methods by 24.4%, requiring only 24.7 seconds and 0.0094 dollars per bug. MemFL thus demonstrates significant improvements by effectively incorporating project-specific knowledge into LLM-based fault localization, delivering high accuracy with reduced time and cost.