A Practical Approach to the Automatic Classification of Security-Relevant Commits

TL;DR

This paper presents a machine learning-based method for automatically classifying security-relevant commits in source code repositories, improving detection accuracy with less training data compared to existing approaches.

Contribution

The authors introduce a novel classification approach that treats code changes as natural language documents, achieving higher precision and better performance with a simpler model.

Findings

High precision of 80% in identifying security-relevant commits

Significant improvement over state-of-the-art methods

Requires less training data and simpler architecture

Abstract

The lack of reliable sources of detailed information on the vulnerabilities of open-source software (OSS) components is a major obstacle to maintaining a secure software supply chain and an effective vulnerability management process. Standard sources of advisories and vulnerability data, such as the National Vulnerability Database (NVD), are known to suffer from poor coverage and inconsistent quality. To reduce our dependency on these sources, we propose an approach that uses machine-learning to analyze source code repositories and to automatically identify commits that are security-relevant (i.e., that are likely to fix a vulnerability). We treat the source code changes introduced by commits as documents written in natural language, classifying them using standard document classification methods. Combining independent classifiers that use information from different facets of commits, our method can yield high precision (80%) while ensuring acceptable recall (43%). In particular, the use of information extracted from the source code changes yields a substantial improvement over the best known approach in state of the art, while requiring a significantly smaller amount of training data and employing a simpler architecture.