LLMs in Code Vulnerability Analysis: A Proof of Concept

TL;DR

This paper investigates the use of large language models for automating software vulnerability detection and repair, demonstrating that fine-tuning enhances performance and that code-specific models excel in complex tasks.

Contribution

It provides a comprehensive evaluation of recent LLMs for vulnerability analysis, comparing fine-tuning and prompt-based methods on benchmark datasets.

Findings

Fine-tuning outperforms zero-shot and few-shot approaches.

Code-specialized models excel in complex tasks.

Current metrics are inadequate for measuring repair quality.

Abstract

Context: Traditional software security analysis methods struggle to keep pace with the scale and complexity of modern codebases, requiring intelligent automation to detect, assess, and remediate vulnerabilities more efficiently and accurately. Objective: This paper explores the incorporation of code-specific and general-purpose Large Language Models (LLMs) to automate critical software security tasks, such as identifying vulnerabilities, predicting severity and access complexity, and generating fixes as a proof of concept. Method: We evaluate five pairs of recent LLMs, including both code-based and general-purpose open-source models, on two recognized C/C++ vulnerability datasets, namely Big-Vul and Vul-Repair. Additionally, we compare fine-tuning and prompt-based approaches. Results: The results show that fine-tuning uniformly outperforms both zero-shot and few-shot approaches across all tasks and models. Notably, code-specialized models excel in zero-shot and few-shot settings on complex tasks, while general-purpose models remain nearly as effective. Discrepancies among CodeBLEU, CodeBERTScore, BLEU, and ChrF highlight the inadequacy of current metrics for measuring repair quality. Conclusions: This study contributes to the software security community by investigating the potential of advanced LLMs to improve vulnerability analysis and remediation.