sec-certs: Examining the security certification practice for better vulnerability mitigation

TL;DR

This paper presents an automated approach to analyze security certification data, revealing how certification practices relate to vulnerability mitigation and identifying factors associated with fewer security issues.

Contribution

We developed unsupervised models and tools to analyze certification documents, linking vulnerabilities to certified products and identifying certification aspects that improve security.

Findings

Automated analysis of certification data is feasible at large scale.

Certain security requirements correlate with fewer vulnerabilities.

Tools can assist in targeted vulnerability mitigation strategies.

Abstract

Products certified under security certification frameworks such as Common Criteria undergo significant scrutiny during the costly certification process. Yet, critical vulnerabilities, including private key recovery (ROCA, Minerva, TPM-Fail...), get discovered in certified products with high assurance levels. Furthermore, assessing which certified products are impacted by such vulnerabilities is complicated due to the large amount of unstructured certification-related data and unclear relationships between the certified products. To address these problems, we conducted a large-scale automated analysis of Common Criteria certificates. We trained unsupervised models to learn which vulnerabilities from NIST's National Vulnerability Database impact existing certified products and how certified products reference each other. Our tooling automates the analysis of tens of thousands of certification-related documents, extracting machine-readable features where manual analysis is unattainable. Further, we identify the security requirements that are associated with products being affected by fewer and less severe vulnerabilities. This indicates which aspects of certification correlate with higher security. We demonstrate how our tool can be used for better vulnerability mitigation on four case studies of known, high-profile vulnerabilities. All tools and continuously updated results are available at https://seccerts.org