A Ground-Truth-Based Evaluation of Vulnerability Detection Across Multiple Ecosystems

TL;DR

This paper presents an empirical evaluation of vulnerability detection tools across multiple ecosystems using a curated ground-truth dataset from OSV, highlighting differences between systems and emphasizing reproducibility.

Contribution

It introduces a systematic, reproducible methodology and an open-source tool for evaluating vulnerability detection across ecosystems using a curated dataset.

Findings

Detection results vary systematically between tools.

The curated dataset enables consistent cross-ecosystem comparison.

Open-source tool supports dataset reconstruction from OSV.

Abstract

Automated vulnerability detection tools are widely used to identify security vulnerabilities in software dependencies. However, the evaluation of such tools remains challenging due to the heterogeneous structure of vulnerability data sources, inconsistent identifier schemes, and ambiguities in version range specifications. In this paper, we present an empirical evaluation of vulnerability detection across multiple software ecosystems using a curated ground-truth dataset derived from the Open Source Vulnerabilities (OSV) database. The dataset explicitly maps vulnerabilities to concrete package versions and enables a systematic comparison of detection results across different tools and services. Since vulnerability databases such as OSV are continuously updated, the dataset used in this study represents a snapshot of the vulnerability landscape at the time of the evaluation. To support reproducibility and future studies, we provide an open-source tool that automatically reconstructs the dataset from the current OSV database using the methodology described in this paper. Our evaluation highlights systematic differences between vulnerability detection systems and demonstrates the importance of transparent dataset construction for reproducible empirical security research.