FGDM: Reasoning Aware Multi-Agentic Framework for Software Bug Detection using Chain of Thought and Tree of Thought Prompting

TL;DR

The paper introduces FGDM, a multi-agent framework utilizing Chain-of-Thought and Tree-of-Thought prompts, combined with flow graphs and a vector database, to improve software bug detection and repair in large codebases.

Contribution

It presents a novel multi-agent framework that enhances bug detection and repair accuracy by integrating flow graphs, advanced prompting techniques, and retrieval of similar past bugs.

Findings

FGDM outperforms existing methods on 100 diverse programs.

Achieved mean reductions of 24.33 and 8.37 in Levenshtein distance for Python and C.

Cosine similarity improved to 0.951 and 0.974 in Python and C.

Abstract

Deep Learning methods are becoming prominent in automated software bug detection; however, they lack the global understanding of the given code. Consequently, their performance tends to degrade, especially when they are applied to large interconnected code bases or complex modular programs. Recently, Large Language Models (LLMs) have proven to be effective at capturing dependencies among multiple interconnected modules in the codebase. This motivated us to propose the Flow-Graph-Driven Multi-Agent Framework (FGDM), which is composed of four agents that operate in a sequential manner. The framework converts the received code to a flow graph, identifies the erroneous segments, and further generates the repaired code. All the employed agents utilize Chain-of-Thought (COT) and Tree-of-Thoughts (TOT) prompts. Additionally, we also integrated with the FAISS vector database to retrieve similar previous bugs and their repairs. We demonstrated the efficacy of the proposed framework over 100 programs from several projects, including Ansible, Black, FastAPI, Keras, Luigi, Matplotlib, Pandas, Scrapy, SpaCy, and Tornado in both C and Python programs. Our experiments demonstrate that the FGDM outperforms the extant approaches and yielded reductions with a mean of 24.33 and 8.37 in Levenshtein distance and similarities of 0.951 and 0.974 in cosine similarity for Python and C, respectively.