Streamlining Security Vulnerability Triage with Large Language Models

TL;DR

This paper introduces CASEY, a novel method using Large Language Models to automate security vulnerability triaging by identifying CWEs and assessing severity, thereby streamlining the process and improving efficiency.

Contribution

The paper presents CASEY, a new approach that leverages prompt engineering with LLMs to automate CWE identification and severity assessment in vulnerability triaging.

Findings

CWE identification accuracy of 68%

Severity assessment accuracy of 73.6%

Combined accuracy of 51.2% for both tasks

Abstract

Bug triaging for security vulnerabilities is a critical part of software maintenance, ensuring that the most pressing vulnerabilities are addressed promptly to safeguard system integrity and user data. However, the process is resource-intensive and comes with challenges, including classifying software vulnerabilities, assessing their severity, and managing a high volume of bug reports. In this paper, we present CASEY, a novel approach that leverages Large Language Models (in our case, the GPT model) that automates the identification of Common Weakness Enumerations (CWEs) of security bugs and assesses their severity. CASEY employs prompt engineering techniques and incorporates contextual information at varying levels of granularity to assist in the bug triaging process. We evaluated CASEY using an augmented version of the National Vulnerability Database (NVD), employing quantitative and qualitative metrics to measure its performance across CWE identification, severity assessment, and their combined analysis. CASEY achieved a CWE identification accuracy of 68%, a severity identification accuracy of 73.6%, and a combined accuracy of 51.2% for identifying both. These results demonstrate the potential of LLMs in identifying CWEs and severity levels, streamlining software vulnerability management, and improving the efficiency of security vulnerability triaging workflows.