A Ground Truth Approach for Assessing Process Mining Techniques

TL;DR

This paper introduces a ground truth-based method for generating realistic synthetic process data with behavioral deviations and errors, enabling more accurate assessment of process mining techniques.

Contribution

It presents a novel approach to create synthetic process logs from models, capturing behavioral deviations and errors, improving evaluation accuracy over existing methods.

Findings

Provides detailed insights into process mining strengths and weaknesses.

Enables quantitative and qualitative assessment of techniques.

Demonstrates effectiveness with three process datasets.

Abstract

The assessment of process mining techniques using real-life data is often compromised by the lack of ground truth knowledge, the presence of non-essential outliers in system behavior and recording errors in event logs. Using synthetically generated data could leverage ground truth for better evaluation. Existing log generation tools inject noise directly into the logs, which does not capture many typical behavioral deviations. Furthermore, the link between the model and the log, which is needed for later assessment, becomes lost. We propose a ground-truth approach for generating process data from either existing or synthetic initial process models, whether automatically generated or hand-made. This approach incorporates patterns of behavioral deviations and recording errors to produce a synthetic yet realistic deviating model and imperfect event log. These, together with the initial model, are required to assess process mining techniques based on ground truth knowledge. We demonstrate this approach to create datasets of synthetic process data for three processes, one of which we used in a conformance checking use case, focusing on the assessment of (relaxed) systemic alignments to expose and explain deviations in modeled and recorded behavior. Our results show that this approach, unlike traditional methods, provides detailed insights into the strengths and weaknesses of process mining techniques, both quantitatively and qualitatively.