Prompting the Priorities: A First Look at Evaluating LLMs for Vulnerability Triage and Prioritization

TL;DR

This study evaluates four large language models using various prompting techniques to assist security analysts in vulnerability triage, revealing moderate effectiveness but emphasizing that LLMs currently cannot replace expert judgment.

Contribution

It provides a comprehensive assessment of LLMs' capabilities in vulnerability prioritization, highlighting their strengths and limitations with specific prompting strategies.

Findings

Gemini outperformed other models on most decision points.

Prompting with exemplars improved model accuracy.

All models tended to over-predict risk.

Abstract

Security analysts face increasing pressure to triage large and complex vulnerability backlogs. Large Language Models (LLMs) offer a potential aid by automating parts of the interpretation process. We evaluate four models (ChatGPT, Claude, Gemini, and DeepSeek) across twelve prompting techniques to interpret semi-structured and unstructured vulnerability information. As a concrete use case, we test each model's ability to predict decision points in the Stakeholder-Specific Vulnerability Categorization (SSVC) framework: Exploitation, Automatable, Technical Impact, and Mission and Wellbeing. Using 384 real-world vulnerabilities from the VulZoo dataset, we issued more than 165,000 queries to assess performance under prompting styles including one-shot, few-shot, and chain-of-thought. We report F1 scores for each SSVC decision point and Cohen's kappa (weighted and unweighted) for the final SSVC decision outcomes. Gemini consistently ranked highest, leading on three of four decision points and yielding the most correct recommendations. Prompting with exemplars generally improved accuracy, although all models struggled on some decision points. Only DeepSeek achieved fair agreement under weighted metrics, and all models tended to over-predict risk. Overall, current LLMs do not replace expert judgment. However, specific LLM and prompt combinations show moderate effectiveness for targeted SSVC decisions. When applied with care, LLMs can support vulnerability prioritization workflows and help security teams respond more efficiently to emerging threats.