Reproducibility in Event-Log Research: A Parametrised Generator and Benchmark for Event-based Signatures

TL;DR

This paper introduces a parametrised generator for synthetic event logs with known signatures, facilitating reproducible benchmarking of signature detection methods in cybersecurity.

Contribution

It presents a novel parametrised generation technique for synthetic event datasets with ground truth signatures, enabling systematic evaluation and comparison of detection approaches.

Findings

DBSCAN achieved over 0.95 Adjusted Rand Index on most datasets

The generator produces realistic, signature-containing datasets for benchmarking

Enhances reproducibility and comparability in cybersecurity research

Abstract

Event-based datasets are crucial for cybersecurity analysis. A key use case is detecting event-based signatures, which represent attacks spanning multiple events and can only be understood once the relevant events are identified and linked. Analysing event datasets is essential for monitoring system security, but their growing volume and frequency create significant scalability and processing difficulties. Researchers rely on these datasets to develop and test techniques for automatically identifying signatures. However, because real datasets are security-sensitive and rarely shared, it becomes difficult to perform meaningful comparative evaluation between different approaches. This work addresses this evaluation limitation by offering a systematic method for generating event logs with known ground truth, enabling reproducible and comparable research. We present a novel parametrised generation technique capable of producing synthetic event datasets that contain event-based signatures for discovery. To demonstrate the capabilities of the technique, we provide a benchmark in signature detection. Our benchmarking demonstrated the suitability of DBSCAN, achieving a score greater than 0.95 Adjusted Rand Index on most generated datasets. This work enhances the ability of researchers to develop and benchmark new cybersecurity techniques, ultimately contributing to more robust and effective cybersecurity measures.