Revisiting Vulnerability Patch Identification on Data in the Wild

TL;DR

This paper evaluates the effectiveness of security patch detection models trained on NVD data when applied in real-world scenarios, revealing significant performance drops and proposing a combined dataset approach for improvement.

Contribution

It demonstrates the limitations of NVD-based training data for in-the-wild patch detection and suggests a hybrid dataset method to enhance model robustness.

Findings

Models trained on NVD data perform poorly on in-the-wild patches.

NVD-linked patches differ significantly from in-the-wild patches in message and content.

Combining NVD data with manually identified patches improves detection robustness.

Abstract

Attacks can exploit zero-day or one-day vulnerabilities that are not publicly disclosed. To detect these vulnerabilities, security researchers monitor development activities in open-source repositories to identify unreported security patches. The sheer volume of commits makes this task infeasible to accomplish manually. Consequently, security patch detectors commonly trained and evaluated on security patches linked from vulnerability reports in the National Vulnerability Database (NVD). In this study, we assess the effectiveness of these detectors when applied in-the-wild. Our results show that models trained on NVD-derived data show substantially decreased performance, with decreases in F1-score of up to 90\% when tested on in-the-wild security patches, rendering them impractical for real-world use. An analysis comparing security patches identified in-the-wild and commits linked from NVD reveals that they can be easily distinguished from each other. Security patches associated with NVD have different distribution of commit messages, vulnerability types, and composition of changes. These differences suggest that NVD may be unsuitable as the \textit{sole} source of data for training models to detect security patches. We find that constructing a dataset that combines security patches from NVD data with a small subset of manually identified security patches can improve model robustness.