Beyond Single Bugs: Benchmarking Large Language Models for Multi-Vulnerability Detection

TL;DR

This paper introduces a comprehensive benchmark for multi-vulnerability detection in code, revealing significant performance degradation of large language models as vulnerability complexity increases across multiple programming languages.

Contribution

It presents a new large-scale dataset and benchmark for multi-vulnerability detection in code, systematically analyzing LLM performance on complex, multi-vulnerability samples across four major languages.

Findings

LLMs' performance drops by up to 40% with increased vulnerability density.

Llama-3.3-70B achieves near-perfect scores on single-vulnerability C tasks.

Python and JavaScript exhibit severe under-counting in complex code samples.

Abstract

Large Language Models (LLMs) have demonstrated significant potential in automated software security, particularly in vulnerability detection. However, existing benchmarks primarily focus on isolated, single-vulnerability samples or function-level classification, failing to reflect the complexity of real-world software where multiple interacting vulnerabilities often coexist within large files. Recent studies indicate that LLMs suffer from "count bias" and "selection bias" in multi-label tasks, yet this has not been rigorously quantified in the domain of code security. In this work, we introduce a comprehensive benchmark for Multi-Vulnerability Detection across four major languages: C, C++, Python, and JavaScript. We construct a dataset of 40,000 files by systematically injecting controlled counts of vulnerabilities (1, 3, 5, and 9) into long-context code samples (7.5k-10k tokens) sourced from CodeParrot. We evaluate five state-of-the-art LLMs, including GPT-4o-mini, Llama-3.3-70B, and the Qwen-2.5 series. Our results reveal a sharp degradation in performance as vulnerability density increases. While Llama-3.3-70B achieves near-perfect F1 scores (approximately 0.97) on single-vulnerability C tasks, performance drops by up to 40% in high-density settings. Notably, Python and JavaScript show distinct failure modes compared to C/C++, with models exhibiting severe "under-counting" (Recall dropping to less than 0.30) in complex Python files.