EvalSVA: Multi-Agent Evaluators for Next-Gen Software Vulnerability Assessment

TL;DR

EvalSVA introduces a multi-agent framework utilizing multiple Large Language Models to autonomously evaluate software vulnerabilities, improving accuracy and providing human-like explanations in vulnerability assessment tasks.

Contribution

The paper presents a novel multi-agent-based framework with diverse communication strategies and a new multilingual dataset for SV assessment, enhancing effectiveness with limited data.

Findings

EvalSVA outperforms previous methods by 44.12% in accuracy.

It achieves 43.29% F1 score in SV assessment.

Provides human-like explanations and detailed reasoning for assessments.

Abstract

Software Vulnerability (SV) assessment is a crucial process of determining different aspects of SVs (e.g., attack vectors and scope) for developers to effectively prioritize efforts in vulnerability mitigation. It presents a challenging and laborious process due to the complexity of SVs and the scarcity of labeled data. To mitigate the above challenges, we introduce EvalSVA, a multi-agent evaluators team to autonomously deliberate and evaluate various aspects of SV assessment. Specifically, we propose a multi-agent-based framework to simulate vulnerability assessment strategies in real-world scenarios, which employs multiple Large Language Models (LLMs) into an integrated group to enhance the effectiveness of SV assessment in the limited data. We also design diverse communication strategies to autonomously discuss and assess different aspects of SV. Furthermore, we construct a multi-lingual SV assessment dataset based on the new standard of CVSS, comprising 699, 888, and 1,310 vulnerability-related commits in C++, Python, and Java, respectively. Our experimental results demonstrate that EvalSVA averagely outperforms the 44.12\% accuracy and 43.29\% F1 for SV assessment compared with the previous methods. It shows that EvalSVA offers a human-like process and generates both reason and answer for SV assessment. EvalSVA can also aid human experts in SV assessment, which provides more explanation and details for SV assessment.