From Lab to Reality: A Practical Evaluation of Deep Learning Models and LLMs for Vulnerability Detection

TL;DR

This paper critically assesses the real-world effectiveness of deep learning and large language models in vulnerability detection, revealing significant performance gaps and emphasizing the need for more robust models and datasets.

Contribution

It provides a systematic evaluation of DL models and LLMs on real-world vulnerability datasets, highlighting their limitations outside benchmark environments.

Findings

Models struggle to differentiate vulnerable from non-vulnerable code in representation space.

Performance drops significantly on out-of-distribution real-world datasets.

Current models have limited generalization across datasets with different distributions.

Abstract

Vulnerability detection methods based on deep learning (DL) have shown strong performance on benchmark datasets, yet their real-world effectiveness remains underexplored. Recent work suggests that both graph neural network (GNN)-based and transformer-based models, including large language models (LLMs), yield promising results when evaluated on curated benchmark datasets. These datasets are typically characterized by consistent data distributions and heuristic or partially noisy labels. In this study, we systematically evaluate two representative DL models-ReVeal and LineVul-across four representative datasets: Juliet, Devign, BigVul, and ICVul. Each model is trained independently on each respective dataset, and their code representations are analyzed using t-SNE to uncover vulnerability related patterns. To assess realistic applicability, we deploy these models along with four pretrained LLMs, Claude 3.5 Sonnet, GPT-o3-mini, GPT-4o, and GPT-5 on a curated dataset, VentiVul, comprising 20 recently (May 2025) fixed vulnerabilities from the Linux kernel. Our experiments reveal that current models struggle to distinguish vulnerable from non-vulnerable code in representation space and generalize poorly across datasets with differing distributions. When evaluated on VentiVul, our newly constructed time-wise out-of-distribution dataset, performance drops sharply, with most models failing to detect vulnerabilities reliably. These results expose a persistent gap between academic benchmarks and real-world deployment, emphasizing the value of our deployment-oriented evaluation framework and the need for more robust code representations and higher-quality datasets.