SecVulEval: Benchmarking LLMs for Real-World C/C++ Vulnerability Detection

TL;DR

This paper introduces SecVulEval, a comprehensive benchmark for evaluating large language models on fine-grained, context-rich C/C++ vulnerability detection at the statement level, addressing limitations of previous datasets.

Contribution

The paper presents SecVulEval, a new benchmark with detailed contextual information and fine-grained labels, enabling more accurate assessment of LLMs in real-world vulnerability detection.

Findings

State-of-the-art LLMs perform poorly on the benchmark, with the best model achieving only 23.83% F1-score.

Rich contextual information improves the evaluation of vulnerability detection models.

Analysis reveals current models lack accurate reasoning in identifying vulnerabilities.

Abstract

Large Language Models (LLMs) have shown promise in software engineering tasks, but evaluating their effectiveness in vulnerability detection is challenging due to the lack of high-quality datasets. Most existing datasets are limited to function-level labels, ignoring finer-grained vulnerability patterns and crucial contextual information. Also, poor data quality such as mislabeling, inconsistent annotations, and duplicates can lead to inflated performance and weak generalization. Moreover, by including only the functions, these datasets miss broader program context, like data/control dependencies and interprocedural interactions, that are essential for accurately understanding real-world security flaws. Without this context, detection models are evaluated under unrealistic assumptions. To address these limitations, this paper introduces SecVulEval, a benchmark designed to support fine-grained evaluation of LLMs and other detection methods with rich contextual information. SecVulEval focuses on real-world C/C++ vulnerabilities at the statement level. This granularity enables more precise evaluation of a model's ability to localize vulnerabilities, beyond simple binary classification at the function level. By incorporating rich contextual information, SecVulEval sets a new standard for vulnerability detection benchmarks in realistic scenarios. This benchmark includes 25,440 function samples covering 5,867 unique CVEs in C/C++ projects from 1999 to 2024. We evaluated the SOTA LLMs with a multi-agent-based approach. The evaluation on our dataset shows that the models are still far from accurately predicting vulnerable statements in a given function. The best-performing Claude-3.7-Sonnet model achieves 23.83% F1-score for detecting vulnerable statements with correct reasoning. Finally, we analyze the LLM outputs and provide insights into their behavior in vulnerability detection for C/C++.