Multi-Granularity Detector for Vulnerability Fixes

TL;DR

MiDas is a multi-granularity neural network ensemble that improves detection of vulnerability-fixing commits in open source software, outperforming existing methods by handling noisy data and imbalanced classes effectively.

Contribution

Introduces MiDas, a novel multi-granularity neural ensemble model that leverages different code change levels to enhance vulnerability fix detection accuracy.

Findings

Outperforms state-of-the-art baseline in AUC by 4.9% (Java) and 13.7% (Python)

Achieves up to 28.2% and 15.9% improvements in EffortCost@L and Popt@L (Java)

Achieves up to 60% and 51.4% improvements in EffortCost@L and Popt@L (Python)

Abstract

With the increasing reliance on Open Source Software, users are exposed to third-party library vulnerabilities. Software Composition Analysis (SCA) tools have been created to alert users of such vulnerabilities. SCA requires the identification of vulnerability-fixing commits. Prior works have proposed methods that can automatically identify such vulnerability-fixing commits. However, identifying such commits is highly challenging, as only a very small minority of commits are vulnerability fixing. Moreover, code changes can be noisy and difficult to analyze. We observe that noise can occur at different levels of detail, making it challenging to detect vulnerability fixes accurately. To address these challenges and boost the effectiveness of prior works, we propose MiDas (Multi-Granularity Detector for Vulnerability Fixes). Unique from prior works, Midas constructs different neural networks for each level of code change granularity, corresponding to commit-level, file-level, hunk-level, and line-level, following their natural organization. It then utilizes an ensemble model that combines all base models to generate the final prediction. This design allows MiDas to better handle the noisy and highly imbalanced nature of vulnerability-fixing commit data. Additionally, to reduce the human effort required to inspect code changes, we have designed an effort-aware adjustment for Midas's outputs based on commit length. The evaluation results demonstrate that MiDas outperforms the current state-of-the-art baseline in terms of AUC by 4.9% and 13.7% on Java and Python-based datasets, respectively. Furthermore, in terms of two effort-aware metrics, EffortCost@L and Popt@L, MiDas also outperforms the state-of-the-art baseline, achieving improvements of up to 28.2% and 15.9% on Java, and 60% and 51.4% on Python, respectively.