Evaluating Conformance Measures in Process Mining using Conformance Propositions (Extended version)

TL;DR

This paper introduces 21 formal conformance propositions to evaluate existing process mining conformance measures, aiming to clarify their properties and foster discussion on their effectiveness.

Contribution

It formulates a comprehensive set of conformance propositions and uses them to systematically evaluate current conformance measures in process mining.

Findings

Identifies gaps and inconsistencies in existing measures

Provides a structured evaluation framework for conformance measures

Stimulates discussion on formal properties of conformance metrics

Abstract

Process mining sheds new light on the relationship between process models and real-life processes. Process discovery can be used to learn process models from event logs. Conformance checking is concerned with quantifying the quality of a business process model in relation to event data that was logged during the execution of the business process. There exist different categories of conformance measures. Recall, also called fitness, is concerned with quantifying how much of the behavior that was observed in the event log fits the process model. Precision is concerned with quantifying how much behavior a process model allows for that was never observed in the event log. Generalization is concerned with quantifying how well a process model generalizes to behavior that is possible in the business process but was never observed in the event log. Many recall, precision, and generalization measures have been developed throughout the years, but they are often defined in an ad-hoc manner without formally defining the desired properties up front. To address these problems, we formulate 21 conformance propositions and we use these propositions to evaluate current and existing conformance measures. The goal is to trigger a discussion by clearly formulating the challenges and requirements (rather than proposing new measures). Additionally, this paper serves as an overview of the conformance checking measures that are available in the process mining area.