Conflicting Scores, Confusing Signals: An Empirical Study of Vulnerability Scoring Systems

TL;DR

This empirical study compares four vulnerability scoring systems using real-world data, revealing significant disparities and highlighting the need for more consistent and transparent vulnerability risk assessments.

Contribution

It provides the first large-scale, outcome-linked empirical comparison of multiple vulnerability scoring systems using real-world data.

Findings

Significant disparities in vulnerability rankings across scoring systems

Scores often do not align with actual exploitation risks

Implications for organizations relying on these metrics for decision-making

Abstract

Accurately assessing software vulnerabilities is essential for effective prioritization and remediation. While various scoring systems exist to support this task, their differing goals, methodologies and outputs often lead to inconsistent prioritization decisions. This work provides the first large-scale, outcome-linked empirical comparison of four publicly available vulnerability scoring systems: the Common Vulnerability Scoring System (CVSS), the Stakeholder-Specific Vulnerability Categorization (SSVC), the Exploit Prediction Scoring System (EPSS), and the Exploitability Index. We use a dataset of 600 real-world vulnerabilities derived from four months of Microsoft's Patch Tuesday disclosures to investigate the relationships between these scores, evaluate how they support vulnerability management task, how these scores categorize vulnerabilities across triage tiers, and assess their ability to capture the real-world exploitation risk. Our findings reveal significant disparities in how scoring systems rank the same vulnerabilities, with implications for organizations relying on these metrics to make data-driven, risk-based decisions. We provide insights into the alignment and divergence of these systems, highlighting the need for more transparent and consistent exploitability, risk, and severity assessments.