RepoAudit: An Autonomous LLM-Agent for Repository-Level Code Auditing

TL;DR

RepoAudit is an autonomous LLM-based agent that efficiently performs repository-level code auditing by analyzing data-flow within functions, reducing hallucinations, and identifying bugs with high precision in real-world projects.

Contribution

This work introduces RepoAudit, the first autonomous LLM agent for repository-level code auditing that combines data-flow analysis, hallucination mitigation, and scalable bug detection.

Findings

Detected 40 true bugs with 78.43% precision in benchmark projects.

Identified 185 new bugs in high-profile repositories, with 174 confirmed or fixed.

Achieved efficient auditing requiring only 0.44 hours and $2.54 per project.

Abstract

Code auditing is the process of reviewing code with the aim of identifying bugs. Large Language Models (LLMs) have demonstrated promising capabilities for this task without requiring compilation, while also supporting user-friendly customization. However, auditing a code repository with LLMs poses significant challenges: limited context windows and hallucinations can degrade the quality of bug reports, and analyzing large-scale repositories incurs substantial time and token costs, hindering efficiency and scalability. This work introduces an LLM-based agent, RepoAudit, designed to perform autonomous repository-level code auditing. Equipped with agent memory, RepoAudit explores the codebase on demand by analyzing data-flow facts along feasible program paths within individual functions. It further incorporates a validator module to mitigate hallucinations by verifying data-flow facts and checking the satisfiability of path conditions associated with potential bugs, thereby reducing false positives. RepoAudit detects 40 true bugs across 15 real-world benchmark projects with a precision of 78.43%, requiring on average only 0.44 hours and $2.54 per project. Also, it detects 185 new bugs in high-profile projects, among which 174 have been confirmed or fixed. We have open-sourced RepoAudit at https://github.com/PurCL/RepoAudit.