VulGuard: An Unified Tool for Evaluating Just-In-Time Vulnerability Prediction Models

TL;DR

VulGuard is an automated, unified tool that simplifies data extraction, model evaluation, and benchmarking for Just-In-Time vulnerability prediction research, enhancing reproducibility and scalability.

Contribution

It introduces VulGuard, a comprehensive framework that automates commit mining, feature extraction, and model evaluation for JIT-VP, supporting scalable and reproducible research.

Findings

Effective in mining and analyzing large repositories

Facilitates comparison of multiple vulnerability prediction models

Proven successful on FFmpeg and Linux kernel projects

Abstract

We present VulGuard, an automated tool designed to streamline the extraction, processing, and analysis of commits from GitHub repositories for Just-In-Time vulnerability prediction (JIT-VP) research. VulGuard automatically mines commit histories, extracts fine-grained code changes, commit messages, and software engineering metrics, and formats them for downstream analysis. In addition, it integrates several state-of-the-art vulnerability prediction models, allowing researchers to train, evaluate, and compare models with minimal setup. By supporting both repository-scale mining and model-level experimentation within a unified framework, VulGuard addresses key challenges in reproducibility and scalability in software security research. VulGuard can also be easily integrated into the CI/CD pipeline. We demonstrate the effectiveness of the tool in two influential open-source projects, FFmpeg and the Linux kernel, highlighting its potential to accelerate real-world JIT-VP research and promote standardized benchmarking. A demo video is available at: https://youtu.be/j96096-pxbs