Detecting Security Fixes in Open-Source Repositories using Static Code Analyzers

TL;DR

This paper explores the use of static code analyzers as features in machine learning models to detect security fixes in open-source software commits, achieving results comparable to current state-of-the-art methods.

Contribution

It demonstrates that static code analyzer features can be effectively used in ML pipelines to identify vulnerability-fixing commits, improving over existing approaches when combined with commit2vec.

Findings

ML models using static code features achieve state-of-the-art results.

Combining static code features with commit2vec improves detection accuracy.

Static code features are useful even when not statistically significant alone.

Abstract

The sources of reliable, code-level information about vulnerabilities that affect open-source software (OSS) are scarce, which hinders a broad adoption of advanced tools that provide code-level detection and assessment of vulnerable OSS dependencies. In this paper, we study the extent to which the output of off-the-shelf static code analyzers can be used as a source of features to represent commits in Machine Learning (ML) applications. In particular, we investigate how such features can be used to construct embeddings and train ML models to automatically identify source code commits that contain vulnerability fixes. We analyze such embeddings for security-relevant and non-security-relevant commits, and we show that, although in isolation they are not different in a statistically significant manner, it is possible to use them to construct a ML pipeline that achieves results comparable with the state of the art. We also found that the combination of our method with commit2vec represents a tangible improvement over the state of the art in the automatic identification of commits that fix vulnerabilities: the ML models we construct and commit2vec are complementary, the former being more generally applicable, albeit not as accurate.