It Only Gets Worse: Revisiting DL-Based Vulnerability Detectors from a Practical Perspective

TL;DR

This paper critically evaluates deep learning-based vulnerability detectors, revealing their limitations in real-world scenarios, and introduces VulTegra, a framework that uncovers key factors influencing detection performance and suggests improvements.

Contribution

The paper presents VulTegra, a comprehensive evaluation framework that compares DL models for vulnerability detection and identifies factors affecting their effectiveness and limitations.

Findings

State-of-the-art detectors have low consistency and limited real-world effectiveness.

Pre-trained models are not always superior to scratch-trained models but have specific strengths.

Adjusting key factors improves recall and F1 scores across multiple detectors.

Abstract

With the growing threat of software vulnerabilities, deep learning (DL)-based detectors have gained popularity for vulnerability detection. However, doubts remain regarding their consistency within declared CWE ranges, real-world effectiveness, and applicability across scenarios. These issues may lead to unreliable detection, high false positives/negatives, and poor adaptability to emerging vulnerabilities. A comprehensive analysis is needed to uncover critical factors affecting detection and guide improvements in model design and deployment. In this paper, we present VulTegra, a novel evaluation framework that conducts a multidimensional comparison of scratch-trained and pre-trained-based DL models for vulnerability detection. VulTegra reveals that state-of-the-art (SOTA) detectors still suffer from low consistency, limited real-world capabilities, and scalability challenges. Contrary to common belief, pre-trained models are not consistently better than scratch-trained models but exhibit distinct strengths in specific contexts.Importantly, our study exposes the limitations of relying solely on CWE-based classification and identifies key factors that significantly affect model performance. Experimental results show that adjusting just one such factor consistently improves recall across all seven evaluated detectors, with six also achieving better F1 scores. Our findings provide deeper insights into model behavior and emphasize the need to consider both vulnerability types and inherent code features for effective detection.