Mitigating Data Imbalance for Software Vulnerability Assessment: Does Data Augmentation Help?

TL;DR

This study investigates the impact of data imbalance on software vulnerability assessment and demonstrates that data augmentation techniques significantly improve model performance across multiple CVSS prediction tasks.

Contribution

The paper is the first large-scale analysis of data imbalance in SV assessment and shows that data augmentation can effectively mitigate this issue, enhancing predictive accuracy.

Findings

Data imbalance significantly affects SV assessment performance.

Simple text augmentation methods outperform baseline models.

Mitigating data imbalance improves model performance by up to 31.8% in MCC.

Abstract

Background: Software Vulnerability (SV) assessment is increasingly adopted to address the ever-increasing volume and complexity of SVs. Data-driven approaches have been widely used to automate SV assessment tasks, particularly the prediction of the Common Vulnerability Scoring System (CVSS) metrics such as exploitability, impact, and severity. SV assessment suffers from the imbalanced distributions of the CVSS classes, but such data imbalance has been hardly understood and addressed in the literature. Aims: We conduct a large-scale study to quantify the impacts of data imbalance and mitigate the issue for SV assessment through the use of data augmentation. Method: We leverage nine data augmentation techniques to balance the class distributions of the CVSS metrics. We then compare the performance of SV assessment models with and without leveraging the augmented data. Results: Through extensive experiments on 180k+ real-world SVs, we show that mitigating data imbalance can significantly improve the predictive performance of models for all the CVSS tasks, by up to 31.8% in Matthews Correlation Coefficient. We also discover that simple text augmentation like combining random text insertion, deletion, and replacement can outperform the baseline across the board. Conclusions: Our study provides the motivation and the first promising step toward tackling data imbalance for effective SV assessment.