On the Performance Analysis of the Adversarial System Variant Approximation Method to Quantify Process Model Generalization

TL;DR

This paper evaluates the Adversarial System Variant Approximation method for process model generalization, revealing its performance under non-ideal conditions and emphasizing the importance of appropriate hyperparameter settings.

Contribution

It provides an experimental analysis of the method's robustness in realistic scenarios, highlighting the need for careful hyperparameter tuning and setting the stage for future research.

Findings

Performance degrades with biased or limited logs

Hyperparameter choice significantly affects accuracy

Method requires awareness of real-world data conditions

Abstract

Process mining algorithms discover a process model from an event log. The resulting process model is supposed to describe all possible event sequences of the underlying system. Generalization is a process model quality dimension of interest. A generalization metric should quantify the extent to which a process model represents the observed event sequences contained in the event log and the unobserved event sequences of the system. Most of the available metrics in the literature cannot properly quantify the generalization of a process model. A recently published method [1] called Adversarial System Variant Approximation leverages Generative Adversarial Networks to approximate the underlying event sequence distribution of a system from an event log. While this method demonstrated performance gains over existing methods in measuring the generalization of process models, its experimental evaluations have been performed under ideal conditions. This paper experimentally investigates the performance of Adversarial System Variant Approximation under non-ideal conditions such as biased and limited event logs. Moreover, experiments are performed to investigate the originally proposed sampling hyperparameter value of the method on its performance to measure the generalization. The results confirm the need to raise awareness about the working conditions of the Adversarial System Variant Approximation method. The outcomes of this paper also serve to initiate future research directions. [1] Theis, Julian, and Houshang Darabi. "Adversarial System Variant Approximation to Quantify Process Model Generalization." IEEE Access 8 (2020): 194410-194427.