Software Vulnerability Prediction in Low-Resource Languages: An Empirical Study of CodeBERT and ChatGPT

TL;DR

This study evaluates the impact of data scarcity on software vulnerability prediction in low-resource languages and explores ChatGPT as a promising solution, showing significant performance improvements over traditional models.

Contribution

It provides the first empirical assessment of ChatGPT's effectiveness for low-resource SV prediction and highlights the limitations of data sampling techniques with CodeBERT.

Findings

CodeBERT's performance drops significantly in low-resource languages.

Data sampling techniques do not improve CodeBERT's predictions.

ChatGPT improves SV prediction accuracy by up to 53.5%.

Abstract

Background: Software Vulnerability (SV) prediction in emerging languages is increasingly important to ensure software security in modern systems. However, these languages usually have limited SV data for developing high-performing prediction models. Aims: We conduct an empirical study to evaluate the impact of SV data scarcity in emerging languages on the state-of-the-art SV prediction model and investigate potential solutions to enhance the performance. Method: We train and test the state-of-the-art model based on CodeBERT with and without data sampling techniques for function-level and line-level SV prediction in three low-resource languages - Kotlin, Swift, and Rust. We also assess the effectiveness of ChatGPT for low-resource SV prediction given its recent success in other domains. Results: Compared to the original work in C/C++ with large data, CodeBERT's performance of function-level and line-level SV prediction significantly declines in low-resource languages, signifying the negative impact of data scarcity. Regarding remediation, data sampling techniques fail to improve CodeBERT; whereas, ChatGPT showcases promising results, substantially enhancing predictive performance by up to 34.4% for the function level and up to 53.5% for the line level. Conclusion: We have highlighted the challenge and made the first promising step for low-resource SV prediction, paving the way for future research in this direction.