Toward Realistic Evaluations of Just-In-Time Vulnerability Prediction

TL;DR

This paper critically evaluates the performance of just-in-time vulnerability prediction methods under realistic, imbalanced dataset conditions, revealing significant performance drops and the ineffectiveness of common imbalance mitigation techniques.

Contribution

It introduces a large-scale, realistic dataset for JIT-VP evaluation and demonstrates the limitations of current methods under real-world class imbalance conditions.

Findings

Performance drops by 98% in real-world settings

Common imbalance techniques are ineffective for JIT-VP

Highlights need for domain-specific solutions in vulnerability prediction

Abstract

Modern software systems are increasingly complex, presenting significant challenges in quality assurance. Just-in-time vulnerability prediction (JIT-VP) is a proactive approach to identifying vulnerable commits and providing early warnings about potential security risks. However, we observe that current JIT-VP evaluations rely on an idealized setting, where the evaluation datasets are artificially balanced, consisting exclusively of vulnerability-introducing and vulnerability-fixing commits. To address this limitation, this study assesses the effectiveness of JIT-VP techniques under a more realistic setting that includes both vulnerability-related and vulnerability-neutral commits. To enable a reliable evaluation, we introduce a large-scale public dataset comprising over one million commits from FFmpeg and the Linux kernel. Our empirical analysis of eight state-of-the-art JIT-VP techniques reveals a significant decline in predictive performance when applied to real-world conditions; for example, the average PR-AUC on Linux drops 98% from 0.805 to 0.016. This discrepancy is mainly attributed to the severe class imbalance in real-world datasets, where vulnerability-introducing commits constitute only a small fraction of all commits. To mitigate this issue, we explore the effectiveness of widely adopted techniques for handling dataset imbalance, including customized loss functions, oversampling, and undersampling. Surprisingly, our experimental results indicate that these techniques are ineffective in addressing the imbalance problem in JIT-VP. These findings underscore the importance of realistic evaluations of JIT-VP and the need for domain-specific techniques to address data imbalance in such scenarios.