SVA-ICL: Improving LLM-based Software Vulnerability Assessment via In-Context Learning and Information Fusion

TL;DR

This paper introduces SVA-ICL, a novel method that enhances large language model performance in software vulnerability assessment by using in-context learning and information fusion of code and vulnerability descriptions.

Contribution

The paper presents a new approach combining in-context learning with multi-modal information fusion to improve LLM-based software vulnerability assessment accuracy.

Findings

SVA-ICL outperforms existing SVA methods in accuracy, F1-score, and MCC.

Component customization significantly impacts SVA-ICL performance.

Effective demonstration selection and fusion ratios are crucial for optimal results.

Abstract

Context: Software vulnerability assessment (SVA) is critical for identifying, evaluating, and prioritizing security weaknesses in software applications. Objective: Despite the increasing application of large language models (LLMs) in various software engineering tasks, their effectiveness in SVA remains underexplored. Method: To address this gap, we introduce a novel approach SVA-ICL, which leverages in-context learning (ICL) to enhance LLM performance. Our approach involves the selection of high-quality demonstrations for ICL through information fusion, incorporating both source code and vulnerability descriptions. For source code, we consider semantic, lexical, and syntactic similarities, while for vulnerability descriptions, we focus on textual similarity. Based on the selected demonstrations, we construct context prompts and consider DeepSeek-V2 as the LLM for SVA-ICL. Results: We evaluate the effectiveness of SVA-ICL using a large-scale dataset comprising 12,071 C/C++ vulnerabilities. Experimental results demonstrate that SVA-ICL outperforms state-of-the-art SVA baselines in terms of Accuracy, F1-score, and MCC measures. Furthermore, ablation studies highlight the significance of component customization in SVA-ICL, such as the number of demonstrations, the demonstration ordering strategy, and the optimal fusion ratio of different modalities. Conclusion: Our findings suggest that leveraging ICL with information fusion can effectively improve the effectiveness of LLM-based SVA, warranting further research in this direction.