VulnLLM-R: Specialized Reasoning LLM with Agent Scaffold for Vulnerability Detection

TL;DR

VulnLLM-R is a specialized reasoning large language model designed for vulnerability detection, outperforming existing static analysis tools and models by reasoning about program states and vulnerabilities, and demonstrating real-world effectiveness.

Contribution

The paper introduces VulnLLM-R, the first specialized reasoning LLM for vulnerability detection, with a novel training recipe and agent scaffold for improved accuracy and real-world application.

Findings

VulnLLM-R outperforms SOTA static analysis tools and reasoning models.

The model detects zero-day vulnerabilities in real-world repositories.

A new training methodology enhances reasoning capabilities in vulnerability detection.

Abstract

We propose VulnLLM-R, the~\emph{first specialized reasoning LLM} for vulnerability detection. Our key insight is that LLMs can reason about program states and analyze the potential vulnerabilities, rather than simple pattern matching. This can improve the model's generalizability and prevent learning shortcuts. However, SOTA reasoning LLMs are typically ultra-large, closed-source, or have limited performance in vulnerability detection. To address this, we propose a novel training recipe with specialized data selection, reasoning data generation, reasoning data filtering and correction, and testing-phase optimization. Using our proposed methodology, we train a reasoning model with seven billion parameters. Through extensive experiments on SOTA datasets across Python, C/C++, and Java, we show that VulnLLM-R has superior effectiveness and efficiency than SOTA static analysis tools and both open-source and commercial large reasoning models. We further conduct a detailed ablation study to validate the key designs in our training recipe. Finally, we construct an agent scaffold around our model and show that it outperforms CodeQL and AFL++ in real-world projects. Our agent further discovers a set of zero-day vulnerabilities in actively maintained repositories. This work represents a pioneering effort to enable real-world, project-level vulnerability detection using AI agents powered by specialized reasoning models. The code is available at~\href{https://github.com/ucsb-mlsec/VulnLLM-R}{github}.