SCoPE: Evaluating LLMs for Software Vulnerability Detection

TL;DR

This paper introduces SCoPE, a framework for processing C/C++ code to improve vulnerability detection using LLMs, refining datasets and evaluating model performance with promising results.

Contribution

SCoPE provides a novel code normalization framework that enhances dataset quality and supports effective vulnerability detection with fine-tuned LLMs.

Findings

SCoPE identified 905 duplicate functions in the dataset.

The best LLM achieved a 53% F1-score in vulnerability detection.

Refined datasets improve model training and evaluation.

Abstract

In recent years, code security has become increasingly important, especially with the rise of interconnected technologies. Detecting vulnerabilities early in the software development process has demonstrated numerous benefits. Consequently, the scientific community started using machine learning for automated detection of source code vulnerabilities. This work explores and refines the CVEFixes dataset, which is commonly used to train models for code-related tasks, specifically the C/C++ subset. To this purpose, the Source Code Processing Engine (SCoPE), a framework composed of strategized techniques that can be used to reduce the size and normalize C/C++ functions is presented. The output generated by SCoPE was used to create a new version of CVEFixes. This refined dataset was then employed in a feature representation analysis to assess the effectiveness of the tool's code processing techniques, consisting of fine-tuning three pre-trained LLMs for software vulnerability detection. The results show that SCoPE successfully helped to identify 905 duplicates within the evaluated subset. The LLM results corroborate with the literature regarding their suitability for software vulnerability detection, with the best model achieving 53% F1-score.