A Practical Solution to Systematically Monitor Inconsistencies in SBOM-based Vulnerability Scanners

TL;DR

This paper introduces SVS-TEST, a method and tool for systematically evaluating the reliability and failure modes of SBOM-based vulnerability scanners, revealing significant inconsistencies and silent failures in real-world tools.

Contribution

The paper presents SVS-TEST, a novel approach for assessing the capability and robustness of SVS-tools using crafted SBOMs and ground truth data.

Findings

Multiple SVS-tools silently fail on valid SBOMs

Significant differences in reliability among SVS-tools

SVS-TEST helps monitor and improve SVS-tool maturity

Abstract

Software Bill of Materials (SBOM) provides new opportunities for automated vulnerability identification in software products. While the industry is adopting SBOM-based Vulnerability Scanning (SVS) to identify vulnerabilities, we increasingly observe inconsistencies and unexpected behavior, that result in false negatives and silent failures. In this work, we present the background necessary to understand the underlying complexity of SVS and introduce SVS-TEST, a method and tool to analyze the capability, maturity, and failure conditions of SVS-tools in real-world scenarios. We showcase the utility of SVS-TEST in a case study evaluating seven real-world SVS-tools using 16 precisely crafted SBOMs and their respective ground truth. Our results unveil significant differences in the reliability and error handling of SVS-tools; multiple SVS-tools silently fail on valid input SBOMs, creating a false sense of security. We conclude our work by highlighting implications for researchers and practitioners, including how organizations and developers of SVS-tools can utilize SVS-TEST to monitor SVS capability and maturity. All results and research artifacts are made publicly available and all findings were disclosed to the SVS-tool developers ahead of time.