Enhancing Pre-Trained Language Models for Vulnerability Detection via Semantic-Preserving Data Augmentation

TL;DR

This paper introduces a semantic-preserving data augmentation method that significantly improves the accuracy of pre-trained language models in vulnerability detection by generating diverse, realistic samples without losing vulnerability semantics.

Contribution

The paper presents a novel natural program transformation technique for data augmentation that enhances vulnerability detection models trained on limited datasets.

Findings

Up to 10.1% increase in accuracy

Up to 23.6% increase in F1 score

Outperforms existing augmentation methods

Abstract

With the rapid development and widespread use of advanced network systems, software vulnerabilities pose a significant threat to secure communications and networking. Learning-based vulnerability detection systems, particularly those leveraging pre-trained language models, have demonstrated significant potential in promptly identifying vulnerabilities in communication networks and reducing the risk of exploitation. However, the shortage of accurately labeled vulnerability datasets hinders further progress in this field. Failing to represent real-world vulnerability data variety and preserve vulnerability semantics, existing augmentation approaches provide limited or even counterproductive contributions to model training. In this paper, we propose a data augmentation technique aimed at enhancing the performance of pre-trained language models for vulnerability detection. Given the vulnerability dataset, our method performs natural semantic-preserving program transformation to generate a large volume of new samples with enriched data diversity and variety. By incorporating our augmented dataset in fine-tuning a series of representative code pre-trained models (i.e., CodeBERT, GraphCodeBERT, UnixCoder, and PDBERT), up to 10.1% increase in accuracy and 23.6% increase in F1 can be achieved in the vulnerability detection task. Comparison results also show that our proposed method can substantially outperform other prominent vulnerability augmentation approaches.