Security Vulnerability Detection with Multitask Self-Instructed Fine-Tuning of Large Language Models

TL;DR

This paper introduces MSIVD, a multitask self-instructed fine-tuning approach combining LLMs and GNNs for improved software vulnerability detection, outperforming existing models on key datasets.

Contribution

The paper presents a novel multitask fine-tuning method integrating LLMs with graph neural networks for vulnerability detection, enhancing accuracy and generalization.

Findings

MSIVD achieves an F1 score of 0.92 on BigVul.

MSIVD outperforms the baseline LineVul.

Training LLMs with GNNs improves vulnerability detection performance.

Abstract

Software security vulnerabilities allow attackers to perform malicious activities to disrupt software operations. Recent Transformer-based language models have significantly advanced vulnerability detection, surpassing the capabilities of static analysis based deep learning models. However, language models trained solely on code tokens do not capture either the explanation of vulnerability type or the data flow structure information of code, both of which are crucial for vulnerability detection. We propose a novel technique that integrates a multitask sequence-to-sequence LLM with pro-gram control flow graphs encoded as a graph neural network to achieve sequence-to-classification vulnerability detection. We introduce MSIVD, multitask self-instructed fine-tuning for vulnerability detection, inspired by chain-of-thought prompting and LLM self-instruction. Our experiments demonstrate that MSIVD achieves superior performance, outperforming the highest LLM-based vulnerability detector baseline (LineVul), with a F1 score of 0.92 on the BigVul dataset, and 0.48 on the PreciseBugs dataset. By training LLMs and GNNs simultaneously using a combination of code and explanatory metrics of a vulnerable program, MSIVD represents a promising direction for advancing LLM-based vulnerability detection that generalizes to unseen data. Based on our findings, we further discuss the necessity for new labelled security vulnerability datasets, as recent LLMs have seen or memorized prior datasets' held-out evaluation data.