PIDSMaker: Building and Evaluating Provenance-based Intrusion Detection Systems

TL;DR

PIDSMaker is an open-source framework that standardizes the development and evaluation of provenance-based intrusion detection systems, improving reproducibility and comparability across different approaches.

Contribution

It introduces a modular, extensible architecture with standardized protocols, enabling consistent evaluation and rapid prototyping of PIDSs.

Findings

Consolidates eight state-of-the-art PIDSs into a unified framework

Provides utilities for ablation, hyperparameter tuning, and visualization

Releases preprocessed datasets and labels for shared evaluation

Abstract

Recent provenance-based intrusion detection systems (PIDSs) have demonstrated strong potential for detecting advanced persistent threats (APTs) by applying machine learning to system provenance graphs. However, evaluating and comparing PIDSs remains difficult: prior work uses inconsistent preprocessing pipelines, non-standard dataset splits, and incompatible ground-truth labeling and metrics. These discrepancies undermine reproducibility, impede fair comparison, and impose substantial re-implementation overhead on researchers. We present PIDSMaker, an open-source framework for developing and evaluating PIDSs under consistent protocols. PIDSMaker consolidates eight state-of-the-art systems into a modular, extensible architecture with standardized preprocessing and ground-truth labels, enabling consistent experiments and apples-to-apples comparisons. A YAML-based configuration interface supports rapid prototyping by composing components across systems without code changes. PIDSMaker also includes utilities for ablation studies, hyperparameter tuning, multi-run instability measurement, and visualization, addressing methodological gaps identified in prior work. We demonstrate PIDSMaker through concrete use cases and release it with preprocessed datasets and labels to support shared evaluation for the PIDS community.