An Empirical Evaluation of LLM-Based Approaches for Code Vulnerability Detection: RAG, SFT, and Dual-Agent Systems

TL;DR

This study compares LLM-based methods for code vulnerability detection, showing that retrieval-augmented generation with external knowledge achieves the highest accuracy, and dual-agent systems improve reasoning transparency.

Contribution

It provides a comprehensive empirical evaluation of RAG, SFT, and dual-agent approaches for vulnerability detection, highlighting the benefits of external knowledge and multi-agent architectures.

Findings

RAG achieved 0.86 accuracy and 0.85 F1 score, outperforming other methods.

SFT with QLoRA adapters showed strong performance.

Dual-Agent system improved reasoning transparency and error mitigation.

Abstract

The rapid advancement of Large Language Models (LLMs) presents new opportunities for automated software vulnerability detection, a crucial task in securing modern codebases. This paper presents a comparative study on the effectiveness of LLM-based techniques for detecting software vulnerabilities. The study evaluates three approaches, Retrieval-Augmented Generation (RAG), Supervised Fine-Tuning (SFT), and a Dual-Agent LLM framework, against a baseline LLM model. A curated dataset was compiled from Big-Vul and real-world code repositories from GitHub, focusing on five critical Common Weakness Enumeration (CWE) categories: CWE-119, CWE-399, CWE-264, CWE-20, and CWE-200. Our RAG approach, which integrated external domain knowledge from the internet and the MITRE CWE database, achieved the highest overall accuracy (0.86) and F1 score (0.85), highlighting the value of contextual augmentation. Our SFT approach, implemented using parameter-efficient QLoRA adapters, also demonstrated strong performance. Our Dual-Agent system, an architecture in which a secondary agent audits and refines the output of the first, showed promise in improving reasoning transparency and error mitigation, with reduced resource overhead. These results emphasize that incorporating a domain expertise mechanism significantly strengthens the practical applicability of LLMs in real-world vulnerability detection tasks.