Automated Discovery of Process Models from Event Logs: Review and Benchmark

TL;DR

This paper systematically reviews and benchmarks automated process discovery methods using open and proprietary event logs, revealing gaps in scalability and performance variability across different quality metrics.

Contribution

It provides the first comprehensive benchmark and comparison of process discovery methods on diverse real-life datasets, highlighting key tradeoffs and research gaps.

Findings

Some methods lack scalability.

Performance varies significantly across metrics.

Divergence in method effectiveness depending on log characteristics.

Abstract

Process mining allows analysts to exploit logs of historical executions of business processes to extract insights regarding the actual performance of these processes. One of the most widely studied process mining operations is automated process discovery. An automated process discovery method takes as input an event log, and produces as output a business process model that captures the control-flow relations between tasks that are observed in or implied by the event log. Various automated process discovery methods have been proposed in the past two decades, striking different tradeoffs between scalability, accuracy and complexity of the resulting models. However, these methods have been evaluated in an ad-hoc manner, employing different datasets, experimental setups, evaluation measures and baselines, often leading to incomparable conclusions and sometimes unreproducible results due to the use of closed datasets. This article provides a systematic review and comparative evaluation of automated process discovery methods, using an open-source benchmark and covering twelve publicly-available real-life event logs, twelve proprietary real-life event logs, and nine quality metrics. The results highlight gaps and unexplored tradeoffs in the field, including the lack of scalability of some methods and a strong divergence in their performance with respect to the different quality metrics used.

Tables

Table 1. TABLE I: Methods assessed by De Weerdt et al.  [ 3 ] (left) and the respective successors (right).

$\alpha$, ${\alpha }^{+}$, ${\alpha }^{++}$ [9, 10, 11]	$\alpha$$ [12]
AGNEs Miner [13]	—
(DT) Genetic Miner [14, 15]	Evolutionary Tree Miner [16]
Heuristics Miner [17, 18]	Heuristics Miner [19, 20, 21, 22]
ILP Miner [23]	Hybrid ILP Miner [24]

Table 2. TABLE III: Descriptive statistics of the public event logs.

Log	Total	Distinct	Total	Distinct	Trace Length
Name	Traces	Traces (%)	Events	Events	min	avg	max
BPIC12	13,087	33.4	262,200	36	3	20	175
BPIC13cp	1,487	12.3	6,660	7	1	4	35
BPIC13inc	7,554	20.0	65,533	13	1	9	123
BPIC14f	41,353	36.1	369,485	9	3	9	167
BPIC151f	902	32.7	21,656	70	5	24	50
BPIC152f	681	61.7	24,678	82	4	36	63
BPIC153f	1,369	60.3	43,786	62	4	32	54
BPIC154f	860	52.4	29,403	65	5	34	54
BPIC155f	975	45.7	30,030	74	4	31	61
BPIC17f	21,861	40.1	714,198	41	11	33	113
RTFMP	150,370	0.2	561,470	11	2	4	20
SEPSIS	1,050	80.6	15,214	16	3	14	185

Table 3. TABLE IV: Descriptive statistics of the proprietary event logs.

Log	Total	Distinct	Total	Distinct	Trace Length
Name	Traces	Traces (%)	Events	Events	min	avg	max
PRT1	12,720	8.1	75,353	9	2	5	64
PRT2	1,182	97.5	46,282	9	12	39	276
PRT3	1,600	19.9	13,720	15	6	8	9
PRT4	20,000	29.7	166,282	11	6	8	36
PRT5	739	0.01	4,434	6	6	6	6
PRT6	744	22.4	6,011	9	7	8	21
PRT7	2,000	6.4	16,353	13	8	8	11
PRT8	225	99.9	9,086	55	2	40	350
PRT9	787,657	0.01	1,808,706	8	1	2	58
PRT10	43,514	0.01	78,864	19	1	1	15
PRT11	174,842	3.0	2,099,835	310	2	12	804
PRT12	37,345	7.5	163,224	20	1	4	27

Table 4. TABLE V: Default parameters evaluation results for the BPIC logs.

	Discovery	Accuracy			Gen.	Complexity				Exec.
Log	Method	Fitness	Precision	F-score	(3-Fold)	Size	CFC	Struct.	Sound	Time(s)
	$\alpha$$	t/o	t/o	t/o	t/o	t/o	t/o	t/o	t/o	t/o
	IM	0.98	0.50	0.66	0.98	59	37	1.00	yes	6.6
	ETM	0.44	0.82	0.57	t/o	67	16	1.00	yes	14,400
BPIC12	FO	-	-	-	-	102	117	0.13	no	9.66
	S-HM6	-	-	-	-	88	46	0.40	no	227.8
	HILP	-	-	-	-	300	460	-	no	772.2
	SM	0.75	0.76	0.76	0.75	53	32	0.72	yes	0.58
	$\alpha$$	-	-	-	-	18	9	-	no	10,112.60
	IM	0.82	1.00	0.90	0.82	9	4	1.00	yes	0.1
	ETM	1.00	0.70	0.82	t/o	38	38	1.00	yes	14,400
BPIC13cp	FO	-	-	-	-	25	23	0.60	no	0.06
	S-HM6	0.94	0.99	0.97	0.94	15	6	1.00	yes	130.0
	HILP	-	-	-	-	10	3	-	yes	0.1
	SM	0.94	0.97	0.96	0.94	12	7	1.00	yes	0.03
	$\alpha$$	0.35	0.91	0.51	t/o	15	7	0.47	yes	4,243.14
	IM	0.92	0.54	0.68	0.92	13	7	1.00	yes	1.0
	ETM	1.00	0.51	0.68	t/o	32	144	1.00	yes	14,400
BPIC13inc	FO	-	-	-	-	43	54	0.77	no	1.41
	S-HM6	0.91	0.96	0.93	0.91	9	4	1.00	yes	0.8
	HILP	-	-	-	-	24	9	-	yes	2.5
	SM	0.91	0.98	0.94	0.91	13	9	1.00	yes	0.23
	$\alpha$$	0.47	0.63	0.54	t/o	62	36	0.31	yes	14,057.48
	IM	0.89	0.64	0.74	0.89	31	18	1.00	yes	3.4
	ETM	0.61	1.00	0.76	t/o	23	9	1.00	yes	14,400
BPIC14f	FO	-	-	-	-	37	46	0.38	no	27.73
	S-HM6	-	-	-	-	202	132	0.73	no	147.4
	HILP	-	-	-	-	80	59	-	no	7.3
	SM	0.76	0.67	0.71	0.76	27	16	0.74	yes	0.59
	$\alpha$$	0.71	0.76	0.73	t/o	219	91	0.22	yes	3,545.9
	IM	0.97	0.57	0.71	0.96	164	108	1.00	yes	0.6
	ETM	0.56	0.94	0.70	t/o	67	19	1.00	yes	14,400
BPIC151f	FO	1.00	0.76	0.87	0.94	146	91	0.25	yes	1.02
	S-HM6	-	-	-	-	204	116	0.56	no	128.1
	HILP	-	-	-	-	282	322	-	no	4.4
	SM	0.90	0.88	0.89	0.90	110	43	0.50	yes	0.48
	$\alpha$$	-	-	-	-	348	164	0.08	no	8,787.48
	IM	0.93	0.56	0.70	0.94	193	123	1.00	yes	0.7
	ETM	0.62	0.91	0.74	t/o	95	32	1.00	yes	14,400
BPIC152f	FO	-	-	-	-	195	159	0.09	no	0.61
	S-HM6	0.98	0.59	0.74	0.97	259	150	0.29	yes	163.2
	HILP	-	-	-	-	-	-	-	-	t/o
	SM	0.77	0.90	0.83	0.77	122	41	0.32	yes	0.25
	$\alpha$$	-	-	-	-	319	169	0.03	no	10,118.15
	IM	0.95	0.55	0.70	0.95	159	108	1.00	yes	1.3
	ETM	0.68	0.88	0.76	t/o	84	29	1.00	yes	14,400
BPIC153f	FO	-	-	-	-	174	164	0.06	no	0.89
	S-HM6	0.95	0.67	0.79	0.95	159	151	0.13	yes	139.9
	HILP	-	-	-	-	433	829	-	no	1,062.9
	SM	0.78	0.94	0.85	0.78	90	29	0.61	yes	0.36
	$\alpha$$	-	-	-	-	272	128	0.13	no	6,410.25
	IM	0.96	0.58	0.73	0.96	162	111	1.00	yes	0.7
	ETM	0.65	0.93	0.77	t/o	83	28	1.00	yes	14,400
BPIC154f	FO	-	-	-	-	157	127	0.14	no	0.50
	S-HM6	0.99	0.64	0.78	0.99	209	137	0.37	yes	136.9
	HILP	-	-	-	-	364	593	-	no	14.7
	SM	0.73	0.91	0.81	0.73	96	31	0.31	yes	0.25
	$\alpha$$	0.62	0.75	0.68	t/o	280	126	0.10	yes	7,603.19
	IM	0.94	0.18	0.30	0.94	134	95	1.00	yes	1.5
	ETM	0.57	0.94	0.71	t/o	88	18	1.00	yes	14,400
BPIC155f	FO	1.00	0.71	0.83	1.00	166	125	0.15	yes	0.56
	S-HM6	1.00	0.70	0.82	1.00	211	135	0.35	yes	141.9
	HILP	-	-	-	-	-	-	-	-	t/o
	SM	0.79	0.94	0.86	0.79	102	30	0.33	yes	0.27
	$\alpha$$	t/o	t/o	t/o	t/o	t/o	t/o	t/o	t/o	t/o
	IM	0.98	0.70	0.82	0.98	35	20	1.00	yes	13.3
	ETM	0.76	1.00	0.86	t/o	42	4	1.00	yes	14,400
BPIC17f	FO	-	-	-	-	98	82	0.25	no	64.33
	S-HM6	0.95	0.62	0.75	0.94	42	13	0.97	yes	143.2
	HILP	-	-	-	-	222	330	-	no	384.5
	SM	0.95	0.85	0.90	0.95	32	17	0.75	yes	2.53

Table 5. TABLE VI: Default parameters evaluation results for the public logs.

	Discovery	Accuracy			Gen.	Complexity				Exec.
Log	Method	Fitness	Precision	F-score	(3-Fold)	Size	CFC	Struct.	Sound?	Time (sec)
	$\alpha$$	t/o	t/o	t/o	t/o	t/o	t/o	t/o	t/o	t/o
	IM	0.99	0.70	0.82	0.99	34	20	1.00	yes	10.9
	ETM	0.99	0.92	0.95	t/o	57	32	1.00	yes	14,400
RTFMP	FO	1.00	0.94	0.97	0.97	31	32	0.19	yes	2.57
	S-HM6	0.98	0.95	0.96	0.98	163	97	1.00	yes	262.7
	HILP	-	-	-	-	57	53	-	no	3.5
	SM	0.99	1.00	1.00	1.00	22	16	0.46	yes	1.25
	$\alpha$$	-	-	-	-	146	156	0.01	no	3,883.12
	IM	0.99	0.45	0.62	0.96	50	32	1.00	yes	0.4
	ETM	0.83	0.66	0.74	t/o	108	101	1.00	yes	14,400
SEPSIS	FO	-	-	-	-	60	63	0.28	no	0.17
	S-HM6	0.92	0.42	0.58	0.92	279	198	1.00	yes	242.7
	HILP	-	-	-	-	87	129	-	no	1.6
	SM	0.73	0.86	0.79	0.73	31	20	0.97	yes	0.05

Table 6. TABLE VII: Default parameters evaluation results for the proprietary logs - 1.

	Discovery	Accuracy			Gen.	Complexity				Exec.
Log	Method	Fitness	Precision	F-score	(3-Fold)	Size	CFC	Struct.	Sound	Time(s)
	$\alpha$$	-	-	-	t/o	45	34	-	no	11,168.54
	IM	0.90	0.67	0.77	0.90	20	9	1.00	yes	2.08
	ETM	0.99	0.81	0.89	t/o	23	12	1.00	yes	14,400
PRT1	FO	-	-	-	-	30	28	0.53	no	0.95
	S-HM6	0.88	0.77	0.82	0.88	59	39	1.00	yes	122.16
	HILP	-	-	-	-	195	271	-	no	2.59
	SM	0.98	0.99	0.98	0.98	29	18	0.79	yes	0.47
	$\alpha$$	-	-	-	-	134	113	0.25	no	3,438.72
	IM	ex	ex	ex	ex	45	33	1.00	yes	1.41
	ETM	0.57	0.94	0.71	t/o	86	21	1.00	yes	14,400
PRT2	FO	-	-	-	-	76	74	0.59	no	0.88
	S-HM6	-	-	-	-	67	105	0.43	no	1.77
	HILP	-	-	-	-	190	299	-	no	21.33
	SM	0.81	0.74	0.77	0.81	40	30	0.85	yes	0.31
	$\alpha$$	0.67	0.76	0.71	0.67	70	40	0.11	yes	220.11
	IM	0.98	0.68	0.80	0.98	37	20	1.00	yes	0.44
	ETM	0.98	0.86	0.92	t/o	51	37	1.00	yes	14,400
PRT3	FO	1.00	0.86	0.92	1.00	34	37	0.32	yes	0.50
	S-HM6	1.00	0.83	0.91	1.00	40	38	0.43	yes	0.67
	HILP	-	-	-	-	343	525	-	no	0.73
	SM	0.83	0.91	0.87	0.88	32	17	0.78	yes	0.17
	$\alpha$$	0.86	0.93	0.90	t/o	21	10	1.00	yes	13,586.48
	IM	0.93	0.75	0.83	0.93	27	13	1.00	yes	1.33
	ETM	0.84	0.85	0.84	t/o	64	28	1.00	yes	14,400
PRT4	FO	-	-	-	-	37	40	0.54	no	6.33
	S-HM6	1.00	0.86	0.93	1.00	370	274	1.00	yes	241.57
	HILP	-	-	-	-	213	306	-	no	5.31
	SM	0.87	0.99	0.93	0.85	34	21	0.65	yes	0.45
	$\alpha$$	1.00	1.00	1.00	1.00	10	1	1.00	yes	2.02
	IM	1.00	1.00	1.00	1.00	10	1	1.00	yes	0.03
	ETM	1.00	1.00	1.00	1.00	10	1	1.00	yes	2.49
PRT5	FO	1.00	1.00	1.00	1.00	10	1	1.00	yes	0.02
	S-HM6	1.00	1.00	1.00	1.00	10	1	1.00	yes	0.11
	HILP	1.00	1.00	1.00	1.00	10	1	1.00	yes	0.05
	SM	1.00	1.00	1.00	1.00	10	1	1.00	yes	0.02
	$\alpha$$	0.80	0.77	0.79	0.80	38	17	0.24	yes	40.10
	IM	0.99	0.82	0.90	0.99	23	10	1.00	yes	2.30
	ETM	0.98	0.80	0.88	t/o	41	16	1.00	yes	14,400
PRT6	FO	1.00	0.91	0.95	1.00	22	17	0.41	yes	0.05
	S-HM6	1.00	0.91	0.95	1.00	22	17	0.41	yes	0.42
	HILP	-	-	-	-	157	214	-	no	0.13
	SM	0.94	1.00	0.97	0.94	16	5	1.00	yes	0.02

Table 7. TABLE VIII: Default parameters evaluation results for the proprietary logs - 2.

	Discovery	Accuracy			Gen.	Complexity				Exec.
Log	Method	Fitness	Precision	F-score	(3-Fold)	Size	CFC	Struct.	Sound?	Time (sec)
	$\alpha$$	0.85	0.90	0.88	0.85	29	9	0.48	yes	143.66
	IM	1.00	0.73	0.84	1.00	29	13	1.00	yes	0.13
	ETM	0.90	0.81	0.85	t/o	60	29	1.00	yes	14,400
PRT7	FO	0.99	1.00	0.99	0.99	26	16	0.39	yes	0.08
	S-HM6	1.00	1.00	1.00	1.00	163	76	1.00	yes	249.74
	HILP	-	-	-	-	278	355	-	no	0.27
	SM	0.91	1.00	0.95	0.92	30	11	0.33	yes	0.06
	$\alpha$$	t/o	t/o	t/o	t/o	t/o	t/o	t/o	t/o	t/o
	IM	0.98	0.33	0.49	0.93	111	92	1.00	yes	0.41
	ETM	0.35	0.88	0.50	t/o	75	12	1.00	yes	14,400
PRT8	FO	-	-	-	-	228	179	0.74	no	0.55
	S-HM6	-	-	-	-	388	323	0.87	no	370.66
	HILP	t/o	t/o	t/o	t/o	t/o	t/o	t/o	t/o	t/o
	SM	0.97	0.67	0.79	0.92	406	488	0.43	yes	1.28
	$\alpha$$	t/o	t/o	t/o	t/o	t/o	t/o	t/o	t/o	t/o
	IM	0.90	0.61	0.73	0.89	28	16	1.00	yes	63.70
	ETM	0.75	0.49	0.59	0.74	27	13	1.00	yes	1,266.71
PRT9	FO	-	-	-	-	32	45	0.72	no	42.83
	S-HM6	0.96	0.98	0.97	0.96	723	558	1.00	yes	318.69
	HILP	-	-	-	-	164	257	-	no	51.47
	SM	0.92	1.00	0.96	0.92	30	20	1.00	yes	9.11
	$\alpha$$	t/o	t/o	t/o	t/o	t/o	t/o	t/o	t/o	t/o
	IM	0.96	0.79	0.87	0.96	41	29	1.00	yes	2.50
	ETM	1.00	0.63	0.77	t/o	61	45	1.00	yes	14,400
PRT10	FO	0.99	0.93	0.96	0.99	52	85	0.64	yes	0.98
	S-HM6	-	-	-	-	77	110	-	no	1.81
	HILP	-	-	-	-	846	3130	-	no	2.55
	SM	0.97	0.97	0.97	0.97	79	68	0.43	yes	0.47
	$\alpha$$	t/o	t/o	t/o	t/o	t/o	t/o	t/o	t/o	t/o
	IM	t/o	t/o	t/o	t/o	549	365	1.00	yes	121.50
	ETM	0.10	1.00	0.18	t/o	21	3	1.00	yes	14,400
PRT11	FO	-	-	-	-	680	713	0.68	no	81.33
	S-HM6	ex	ex	ex	ex	ex	ex	ex	ex	ex
	HILP	t/o	t/o	t/o	t/o	t/o	t/o	t/o	t/o	t/o
	SM	-	-	-	-	712	609	0.12	no	19.53
	$\alpha$$	t/o	t/o	t/o	t/o	t/o	t/o	t/o	t/o	t/o
	IM	1.00	0.77	0.87	1.00	32	25	1.00	yes	3.94
	ETM	0.63	1.00	0.77	t/o	21	8	1.00	yes	14,400
PRT12	FO	-	-	-	-	87	129	0.38	no	1.67
	S-HM6	-	-	-	-	4370	3191	1.00	yes	347.57
	HILP	-	-	-	-	926	2492	-	no	7.34
	SM	0.96	0.99	0.97	0.95	97	84	0.58	yes	0.36

Table 8. TABLE IX: Hyperparameters-optimization evaluation results.

Log	Metrics	IM	FO	S-HM6	SM
	Fitness	0.81	0.82	0.96	0.91
BPIC12	Precision	0.64	0.41	0.66	0.83
	F-score	0.71	0.54	0.78	0.87
	Fitness	0.99	-	0.96	0.94
BPIC13cp	Precision	0.98	-	1.00	0.97
	F-score	0.98	-	0.98	0.96
	Fitness	1.00	-	0.93	0.91
BPIC13inc	Precision	0.71	-	0.98	0.98
	F-score	0.83	-	0.96	0.94
	Fitness	0.75	0.94	0.91	0.80
BPIC14f	Precision	0.97	0.85	0.84	0.99
	F-score	0.85	0.89	0.88	0.89
	Fitness	1.00	0.90	0.88	0.95
BPIC151f	Precision	0.57	0.88	0.89	0.86
	F-score	0.72	0.89	0.89	0.90
	Fitness	0.69	0.99	0.99	0.81
BPIC152f	Precision	0.79	0.63	0.62	0.86
	F-score	0.74	0.77	0.76	0.83
	Fitness	0.77	0.80	0.81	0.78
BPIC153f	Precision	0.80	0.86	0.77	0.94
	F-score	0.79	0.83	0.79	0.85
	Fitness	0.73	0.76	0.99	0.77
BPIC154f	Precision	0.87	0.87	0.66	0.90
	F-score	0.80	0.81	0.79	0.83
	Fitness	0.65	0.81	0.82	0.86
BPIC155f	Precision	0.87	0.91	0.94	0.90
	F-score	0.75	0.86	0.87	0.88
	Fitness	1.00	1.00	0.97	0.95
BPIC17f	Precision	0.70	0.07	0.70	0.85
	F-score	0.82	0.12	0.81	0.90
	Fitness	0.96	1.00	0.98	0.99
RTFMP	Precision	0.72	0.94	0.96	1.00
	F-score	0.82	0.97	0.97	1.00
	Fitness	0.66	0.74	0.92	0.85
SEPSIS	Precision	0.91	0.67	0.42	0.73
	F-score	0.76	0.70	0.58	0.79
	Fitness	1.00	0.98	0.96	0.98
PRT1	Precision	0.82	0.92	0.98	0.99
	F-score	0.90	0.95	0.97	0.98
	Fitness	-	1.00	-	0.81
PRT2	Precision	-	0.17	-	0.74
	F-score	-	0.30	-	0.77
	Fitness	0.87	1.00	0.99	1.00
PRT3	Precision	0.93	0.86	0.85	0.92
	F-score	0.90	0.92	0.91	0.96
	Fitness	0.86	1.00	0.93	0.99
PRT4	Precision	1.00	0.87	0.96	1.00
	F-score	0.92	0.93	0.95	0.99
	Fitness	1.00	1.00	1.00	1.00
PRT5	Precision	1.00	1.00	1.00	1.00
	F-score	1.00	1.00	1.00	1.00
	Fitness	0.92	1.00	0.98	0.94
PRT6	Precision	1.00	0.91	0.96	1.00
	F-score	0.96	0.95	0.97	0.97
	Fitness	0.88	0.99	1.00	0.93
PRT7	Precision	1.00	1.00	1.00	1.00
	F-score	0.93	0.99	1.00	0.96
	Fitness	0.79	1.00	0.93	0.99
PRT8	Precision	0.37	0.14	0.42	0.66
	F-score	0.51	0.25	0.58	0.79
	Fitness	0.93	-	0.96	0.99
PRT9	Precision	0.68	-	0.98	1.00
	F-score	0.78	-	0.97	0.99
	Fitness	0.99	0.99	0.98	0.99
PRT10	Precision	0.79	0.93	0.83	0.97
	F-score	0.88	0.96	0.90	0.98
	Fitness	ex	-	-	-
PRT11	Precision	ex	-	-	-
	F-score	ex	-	-	-
	Fitness	0.97	1.00	-	0.98
PRT12	Precision	0.85	0.80	-	0.97
	F-score	0.91	0.89	-	0.98

Full text

1 Introduction

Modern information systems maintain detailed trails of the business processes they support, including records of key process execution events, such as the creation of a case or the execution of a task within an ongoing case. Process mining techniques allow analysts to extract insights about the actual performance of a process from collections of such event records, also known as event logs [1]. In this context, an event log consists of a set of traces, each trace itself consisting of the sequence of events related to a given case.

One of the most widely investigated process mining operations is automated process discovery. An automated process discovery method takes as input an event log, and produces as output a business process model that captures the control-flow relations between tasks that are observed in or implied by the event log.

In order to be useful, such automatically discovered process models must accurately reflect the behavior recorded in or implied by the log. Specifically, the process model discovered from an event log should be able to: (i) generate each trace in the log, or for each trace in the log, generate a trace that is similar to it; (ii) generate traces that are not in the log but that are identical or similar to traces of the process that produced the log; and (iii) not generate other traces [2]. The first property is called fitness, the second generalization and the third precision. In addition, the discovered process model should be as simple as possible, a property that is usually quantified via complexity measures.

The problem of automated discovery of process models from event logs has been intensively researched in the past two decades. Despite a rich set of proposals, state-of-the-art automated process discovery methods suffer from two recurrent deficiencies when applied to real-life logs [3]: (i) they produce large and spaghetti-like models; and (ii) they produce models that either poorly fit the event log (low fitness) or over-generalize it (low precision or low generalization). Striking a tradeoff between these quality dimensions in a robust manner has proved to be a difficult problem.

So far, automated process discovery methods have been evaluated in an ad hoc manner, with different authors employing different datasets, experimental setups, evaluation measures and baselines, often leading to incomparable conclusions and sometimes unreproducible results due to the use of non-publicly available datasets. This work aims at filling this gap by providing: (i) a systematic review of automated process discovery methods; and (ii) a comparative evaluation of seven implementations of representative methods, using an open-source benchmark framework and covering twelve publicly-available real-life event logs, twelve proprietary real-life event logs, and nine quality metrics covering all four dimensions mentioned above (fitness, precision, generalization and complexity), as well as execution time.

The outcomes of this research are a classified inventory of automated process discovery methods and a benchmark designed to enable researchers to empirically compare new automated process discovery methods against existing ones in a unified setting. The benchmark is provided as an open-source command-line Java application to enable researchers to replicate the reported experiments with minimal configuration effort.

The rest of the article is structured as follows. Section 2 describes the search protocol used for the systematic literature review, whereas Section 3 classifies the methods identified in the review. Then, Section 4 introduces the experimental benchmark and results, whereas Section 5 discusses the overall findings and Section 6 acknowledges the threats to the validity of the study. Finally, Section 7 relates this work to previous reviews and comparative studies in the field and Section 8 concludes the paper and outlines future work directions.

2 Search protocol

In order to identify and classify research in the area of automated process discovery, we conducted a Systematic Literature Review (SLR) through a scientific, rigorous and replicable approach as specified by Kitchenham [4].

First, we formulated a set of research questions to scope the search, and developed a list of search strings. Next, we ran the search strings on different data sources. Finally, we applied inclusion criteria to select the studies retrieved through the search.

2.1 Research questions

The objective of our SLR is to analyse research studies related to automated (business) process discovery from event logs. This means, for example, that methods performing only trace clustering or log-filtering are not considered in our analysis. To this aim, we formulated the following research questions:

• RQ1

What methods exist for automated (business) process discovery from event logs?

• RQ2

What type of process models can be discovered by these methods, and in which modeling language?

• RQ3

Which semantic can be captured by a model discovered by these methods?

• RQ4

What tools are available to support these methods?

• RQ5

What type of data has been used to evaluate these methods, and from which application domains?

RQ1 is the core research question, which aims at identifying existing methods to perform (business) process discovery from event logs. The other questions allow us to identify a set of classification criteria. Specifically, RQ2 categorizes the output of a method on the basis of the type of process model discovered (i.e., procedural, declarative or hybrid), and the specific modeling language employed (e.g., Petri nets, BPMN, Declare). RQ3 delves into the specific semantic constructs supported by a method (e.g., exclusive choice, parallelism, loops). RQ4 explores what tool support the different methods have, while RQ5 investigates how the methods have been evaluated and in which application domains.

2.2 Search string development and validation

Next, we developed four search strings by deriving keywords from our knowledge of the subject matter. We first determined that the term “process discovery” is a very generic term which would allow us to retrieve the majority of methods in this area. Furthermore, we used “learning” and “workflow” as synonyms of “discovery” and “process” (respectively). This led to the following four search strings: (i) “process discovery”, (ii) “workflow discovery”, (iii) “process learning”, (iv) “workflow learning”. We intentionally excluded the terms “automated” and “automating” in the search strings, because these terms are often not explicitly used.

However, this led to retrieving many more studies than those that actually focus on automated process discovery, e.g., studies on process discovery via workshops or interviews. Thus, if a query on a specific data source returned more than one thousand results, we refined it by combining the selected search string with the term “business” or “process mining” to obtain more focused results, e.g., “process discovery AND process mining” or “process discovery AND business”. According to this criterion, the final search strings used for our search were the following:

• i.

“process discovery AND process mining”

• ii.

“process learning AND process mining”

• iii.

“workflow discovery”

• iv.

“workflow learning”

First, we applied each of the four search strings to Google Scholar, retrieving studies based on the occurrence of one of the search strings in the title, the keywords or the abstract of a paper. Then, we used the following six popular academic databases: Scopus, Web of Science, IEEE Xplore, ACM Digital Library, SpringerLink, ScienceDirect, to double check the studies retrieved from Google Scholar. We noted that this additional search did not return any relevant study that was not already discovered in our primary search. The search was completed in December 2017.

2.3 Study selection

As a last step, as suggested by [5, 6, 7, 8], we defined inclusion criteria to ensure an unbiased selection of relevant studies. To be retained, a study must satisfy all the following inclusion criteria.

• IN1

The study proposes a method for automated (business) process discovery from event logs. This criterion draws the borders of our search scope and it is direct consequence of RQ1.

• IN2

The study proposes a method that has been implemented and evaluated. This criterion let us exclude methods whose properties have not been evaluated nor analyzed.

• IN3

The study is published in 2011 or later. Earlier studies have been reviewed and evaluated by De Weerdt et al.[3], therefore, we decided to focus only on the successive studies. Nevertheless, we performed a mapping of the studies assessed in 2011 [3] and their successors (when applicable), cf. Table I.

• IN4

The study is peer-reviewed. This criterion guarantees a minimum reasonable quality of the studies included in this SLR.

• IN5

The study is written in English.

Inclusion criteria IN3, IN4 and IN5 were automatically applied through the configuration of the search engines. After the application of the latter three inclusion criteria, we obtained a total of 2,820 studies. Then, we skimmed title and abstract of these studies to exclude those studies that were clearly not compliant with IN1. As a result of this first iteration, we obtained 344 studies.

Then, we assessed each of the 344 studies against the inclusion criteria IN1 and IN2. The (combined) assessment of IN1 and IN2 on the 344 selected studies was performed independently by two authors of this paper, whose decisions were compared in order to resolve inconsistencies with the mediation of a third author, when needed. The assessment of IN1 was based on the accurate reading of the abstract, introduction and conclusion. Whilst, to determine whether a study fulfilled IN2, we relied on the findings reported in the evaluation of the studies. As result of the iterations, we found 86 studies matching the five inclusion criteria.

However, many of these studies refer to the same automated process discovery method, i.e., some studies are either extensions, optimization, preliminaries or generalization of another study. For such reason, we decided to group the studies by either the last version or the most general one. When in doubt, the grouping decision was taken after a consultation with the main author. At the end of this process, as shown in Table II, 35 main groups of discovery algorithms were identified.

The excel sheet available at https://drive.google.com/open?id=1fW8WLXSwI2ntiPu3XVgsDI1cJUJr762G reports the 344 studies found after the first iteration. For each of these studies, we explicitly refer to the inclusion criterion fulfilled for the study to be selected. Furthermore, each selected study has a reference to the group it belongs to (unless it is the main study).

Figure 1 shows how the studies are distributed over time. We can see that the interest in the topic of automated process discovery grew over time with a sharp increase between 2013 and 2014, and lately declining close to the average number of studies per year.

3 Classification of methods

Driven by the research questions defined in Section 2.1, we identified the following classification dimensions to survey the methods described in the primary studies:

Model type (procedural, declarative, hybrid) and model language (e.g., Petri nets, BPMN, Declare) — RQ2 2. 2.

Semantic captured in procedural models: parallelism (AND), exclusive choice (XOR), inclusive choice (OR), loop — RQ3 3. 3.

Type of implementation (standalone or plug-in, and tool accessibility) — RQ4 4. 4.

Type of evaluation data (real-life, synthetic or artificial log, where a synthetic log is one generated from a real-life model while an artificial log is one generated from an artificial model) and application domain (e.g., insurance, banking, healthcare) — RQ5.

This information is summarized in Table II. Each entry in this table refers to the main study of the 35 groups found. Also, we cited all the studies that relate to the main one. Collectively, the information reported by Table II allows us to answer the first research question: “what methods exist for automated process discovery?”

In the remainder of this section, we proceed with surveying each main study method along the above classification dimensions, to answer the other research questions.

3.1 Model type and language (RQ2)

The majority of methods (26 out of 35) produce procedural models. Six approaches [26, 33, 46, 49, 75, 85] discover declarative models in the form of Declare constraints, whereas [56] produces declarative models using the WoMan formalism. The methods in [62, 68] are able to discover hybrid models as a combination of Petri nets and Declare constraints.

Regarding the modeling languages of the discovered process model, we notice that Petri nets is the predominant one. However, more recently, we have seen the appearance of methods that produce models in BPMN, a language that is more practically-oriented and less technical than Petri nets. This denotes a shift in the target audience of these methods, from data scientists to practitioners, such as business analysts and decision managers. Other technical languages employed, besides Petri nets, include Causal nets, State machines and simple Directed Acyclic Graphs, while Declare is the most commonly-used language when producing declarative models.

Petri nets. In [25], the authors describe an algorithm to extract block-structured Petri nets from event logs. The algorithm works by first building an adjacency matrix between all pairs of tasks and then analyzing the information in it to extract block-structured models consisting of basic sequence, choice, parallel, loop, optional and self-loop structures as building blocks. The method has been implemented in a standalone tool called HK.

The method presented in [12] is based on the \alpha\$$ algorithm, which can discover invisible tasks involved in non-free-choice constructs. The algorithm is an extension of the well-known \alpha$ algorithm, one of the very first algorithms for automated process discovery, originally presented in [1].

In [96], the authors propose a generic divide-and-conquer framework for the discovery of process models from large event logs. The method allows to part the event log in smaller logs and discover a model from each of them. The output is then assembled from all the models discovered from the sublogs. The method aims to produce high quality models reducing overall the complexity. The preliminary studies [97, 98, 99, 100, 101] widely illustrate the idea of splitting a large event log into a collection of smaller logs to improve the performance of a discovery algorithm.

van Zelst et al. [24, 102, 103] propose an improvement of the ILP miner implemented in [23], their method is based on hybrid variable-based regions. Through hybrid variable-based regions, it is possible to vary the number of variables used within the ILP problems being solved. Using a different number of variables has an impact on the average computation time for solving the ILP problem.

In [77, 78], the authors propose an approach that allows the discovery of Petri nets using the theory of grammatical inference. The method has been implemented as a standalone application called RegPFA.

The approach proposed in [87] is based on the observation that activities with no dependencies in an event log can be executed in parallel. In this way, this method can discover process models with concurrency even if the logs fail to meet the completeness criteria. The method has been implemented in a tool called ProM-D.

In [55], the authors propose the use of numerical abstract domains for discovering Petri nets from large logs while guaranteeing formal properties of the discovered models. The technique guarantees the discovery of Petri nets that can reproduce every trace in the log and that are minimal in describing the log behavior.

The approach introduced in [88] addresses the problem of discovering sound Workflow nets from incomplete logs. The method is based on the concept of invariant occurrence between activities, which is used to identify sets of activities (named conjoint occurrence classes) that can be used to infer the behaviors not exhibited in the log.

In [83], the authors leverage data carried by tokens in the execution of a business process to track the state changes in the so-called token logs. This information is used to improve the performance of standard discovery algorithms.

Process trees. The Inductive Miner [38] and the Evolutionary Tree Miner [16] are both based on the extraction of process trees from an event log. Concerning the former, many different variants have been proposed during the last years, but its first appearance is in [39]. Successively, since that method was unable to deal with infrequent behavior, an improvement was proposed in [38], which efficiently drops infrequent behavior from logs, still ensuring that the discovered model is behaviorally correct (sound) and highly fitting. Another variant of the Inductive Miner is presented in [40]. This variant can minimize the impact of incompleteness of the input logs. In [42], the authors discuss ways of systematically treating lifecycle information in the discovery task. They introduce a process discovery technique that is able to handle lifecycle data to distinguish between concurrency and interleaving. The method proposed in [41] provides a graphical support for navigating the discovered model and the one described in [45] can deal with cancelation or error-handling behaviors (i.e., with logs containing traces that do not complete normally). Finally, the variant presented in [43] and [44] combines scalability with quality guarantees. It can be used to mine large event logs and produces sound models.

In [16], Buijs et al. introduce the Evolutionary Tree Miner. This method is based on a genetic algorithm that allows the user to drive the discovery process based on preferences with respect to the four quality dimensions of the discovered model: fitness, precision, generalization and complexity. The importance of these four dimensions and how to address their balance in process discovery is widely discussed in the related studies[50, 51, 52, 53, 54].

Causal nets. Greco et al. propose a discovery method that returns causal nets [69, 70]. A causal net is a net where only the causal relation between activities in a log is represented. The proposed method encodes causal relations gathered from an event log and if available, background knowledge in terms of precedence constraints over the topology of the resulting process models. A discovery algorithm is formulated in terms of reasoning problems over precedence constraints.

In [71], the authors propose a method for automated process discovery using Maximal Pattern Mining where they discover recurrent sequences of events in the traces of the log. Starting from these patterns they build process models in the form of causal nets.

ProDiGen, a standalone miner by Vazquez et al.[73, 74], allows users to discover causal nets from event logs using a genetic algorithm. The algorithm is based on a fitness function that takes into account completeness, precision and complexity and specific crossover and mutation operators.

Another method that produces causal nets is the Proximity Miner, presented in [89, 90]. This method extracts behavioral relations between the events of the log which are then enhanced using inputs from domain experts.

Finally, in [95], the authors propose a method to discover causal nets that optimizes the scalability and interpretability of the outputs. The process under analysis is decomposed into a set of stages, such that each stage can be mined separately. The proposed technique discovers a stage decomposition that maximizes modularity.

State machines. The CSM Miner, discussed in [81, 82], discovers state machines from event logs. Instead of focusing on the events or activities that are executed in the context of a particular process, this method concentrates on the states of the different process perspectives and discover how they are related with each other. These relations are expressed in terms of Composite State Machines. The CSM Miner provides an interactive visualization of these multi-perspective state-based models.

BPMN models. In [80], Conforti et al. present the BPMN Miner, a method for the automated discovery of BPMN models containing sub-processes, activity markers such as multi-instance and loops, and interrupting and non-interrupting boundary events (to model exception handling). The method has been subsequently improved in [79] to make it robust to noise in event logs.

Another method to discover BPMN models has been presented in [72]. In this approach, a hierarchical view on process models is formally specified and an evolution strategy is applied on it. The evolution strategy, which is guided by the diversity of the process model population, efficiently finds the process models that best represent a given event log.

A further method to discover BPMN models is the Dynamic Constructs Competition Miner [63, 65, 66]. This method extends the Constructs Competition Miner presented in [64]. The method is based on a divide-and-conquer algorithm which discovers block-structured process models from logs.

In [92], the authors propose a discovery method that produces simple process models with low branching complexity and consistently high and balanced fitness, precision and generalization. The approach combines a technique to filter the directly-follows graph induced by an event log, with an approach to identify combinations of split gateways that accurately capture the concurrency, conflict and causal relations between neighbors in the directly-follows graph.

Fodina [93, 94] is a discovery method based on the Heuristics Miner [18]. However, differently from the Heuristics Miner, Fodina is more robust to noisy data, is able to discover duplicate activities, and allows for flexible configuration options to drive the discovery according to end user inputs.

In [21], the authors present the Flexible Heuristics Miner. This method can discover process models containing non-trivial constructs but with a low degree of block structuredness. At the same time, the method can cope well with noise in event logs. The discovered models are a specific type of Heuristics nets where the semantics of splits and joins is represented using split/join frequency tables. This results in easy to understand process models even in the presence of non-trivial constructs and log noise. The discovery algorithm is based on that of the original Heuristics Miner method [18]. In [22], the method presented in [21] has been improved as anomalies were found concerning the validity and completeness of the resulting process model. The improvements have been implemented in the Updated Heuristics Miner. A data-aware version of the Heuristics Miner that takes into consideration data attached to events in a log has been presented in [20]. Finally, in [19, 91], the authors propose an improvement of the Heuristics Miner algorithm to separate the objective of producing accurate models and that of ensuring their structuredness and soundness. Instead of directly discovering a structured process model, the approach first discovers accurate, possibly unstructured (and unsound) process models, and then transforms the resulting process model into a structured (and sound) one.

Declarative models. In [27], the authors present the first basic approach for mining declarative process models expressed using Declare constraints [104, 105]. This approach was improved in [26] using a two-phase approach. The first phase is based on an apriori algorithm used to identify frequent sets of correlated activities. A list of candidate constraints is built on the basis of the correlated activity sets. In the second phase, the constraints are checked by replaying the log on specific automata, each accepting only those traces that are compliant to one constraint. Those constraints satisfied by a percentage of traces higher than a user-defined threshold, are discovered. Other variants of the same approach are presented in [28, 29, 30, 31, 32]. The technique presented in [28] leverages apriori knowledge to guide the discovery task. In [29], the approach is adapted to be used in cross-organizational environments in which different organizations execute the same process in different variants. In [30], the author extends the approach to discover metric temporal constraints, i.e., constraints taking into account the time distance between events. Finally, in [31, 32], the authors propose mechanisms to reduce the execution times of the original approach presented in [26].

MINERful [33, 34, 35] discovers Declare constraints using a two-phase approach. The first phase computes statistical data describing the occurrences of activities and their interplay in the log. The second one checks the validity of Declare constraints by querying such a statistic data structure (knowledge base). In [36, 37], the approach is extended to discover target-branched Declare constraints, i.e., constraints in which the target parameter is the disjunction of two or more activities.

The approach presented in [46] is the first approach for the discovery of Declare constraints with an extended semantics that take into consideration data conditions. The data-aware semantics of Declare presented in this paper is based on first-order temporal logic. The method presented in [75, 76] is based on the use of discriminative rule mining to determine how the characteristics of the activity lifecycles in a business process influence the validity of a Declare constraint in that process.

Other approaches for the discovery of Declare constraints have been presented in [49, 85]. In [49], the authors present the Evolutionary Declare Miner that implements the discovery task using a genetic algorithm. The SQLMiner, presented in [85], is based on a mining approach that directly works on relational event data by querying a log with standard SQL. By leveraging database performance technology, the mining procedure is extremely fast. Queries can be customized and cover process perspectives beyond control flow[86].

The WoMan framework, proposed by Ferilli in [56] and further extended in the related studies [57, 58, 59, 60, 61], includes a method to learn and refine process models from event logs, by discovering first-order logic constraints. It guarantees incrementality in learning and adapting the models, the ability to express triggers and conditions on the process tasks and efficiency.

Further approaches. In [47, 48], the authors introduce a monitoring framework for automated process discovery. A monitoring context is used to extract traces from relational event data and attach different types of metrics to them. Based on these metrics, traces with certain characteristics can be selected and used for the discovery of process models expressed as directly-follows graphs.

Vasilecas et al.  [67] present a method for the extraction of directed acyclic graphs from event logs. Starting from these graphs, they generate Bayesian belief networks, one of the most common probabilistic models, and use these networks to efficiently analyze business processes.

In [84], the authors show how conditional partial order graphs, a compact representation of families of partial orders, can be used for addressing the problem of compact and easy-to-comprehend representation of event logs with data. They present algorithms for extracting both the control flow as well as relevant data parameters from a given event log and show how conditional partial order graphs can be used to visualize the obtained results. The method has been implemented as a Workcraft plug-in and as a standalone application called PGminer.

The Hybrid Miner, presented in [62], puts forward the idea of discovering a hybrid model from an event log based on the semantics defined in [106]. According to such semantics, a hybrid process model is a hierarchical model, where each node represents a sub-process, which may be specified in a declarative or procedural way. Petri nets are used for representing procedural sub-processes and Declare for representing declarative sub-processes.

[68] proposes an approach for the discovery of hybrid models based on the semantics proposed in [107]. Differently from the semantics introduced in [106], where a hybrid process model is hierarchical, the semantics defined in [107] is devoted to obtain a fully mixed language, where procedural and declarative constructs can be connected with each other.

3.2 Procedural language constructs (RQ3)

All the 26 methods that discover a procedural model can detect the basic control-flow structure of sequence. Out of these methods, only four can also discover inclusive choices, but none in the context of non-block-structured models. In fact, [38, 16] are able to directly identify block-structured inclusive choices (using process trees), while [96, 79] can detect this construct only when used on top of the methods in [16] or [38] (i.e., indirectly).

The remaining 22 methods can discover constructs for parallelism, exclusive choice and loops, with the exception of [47], which can detect exclusive choice and loops but not parallelism, [84], which can detect parallelism and exclusive choice but not loops, and [67], which can discover exclusive choices only.

3.3 Implementation (RQ4)

Over 50% of the methods (19 out of 35) provide an implementation as a plug-in for the ProM platform. 111http://promtools.org The reason behind the popularity of ProM can be explained by its open-source and portable framework, which allows researchers to easily develop and test new discovery algorithms. Also, ProM is the first software tool for process mining. One of the methods which has a ProM implementation [33] is also available as standalone tool. The works [79, 92, 95, 19] provide both a standalone implementation and a further implementation as a plug-in for Apromore.222http://apromore.org Apromore is an online process analytics platform, also open source, and has a growing consensus among academics as a process mining tool oriented towards end users. Finally, one method [84] has been implemented as a plug-in for Workcraft, 333http://workcraft.org a platform for designing concurrent systems.

Notice that 22 tools out of 35 are made publicly available to the community. These exclude 4 ProM plug-ins and 9 standalone tools.

3.4 Evaluation data and domains (RQ5)

The surveyed methods have been evaluated using three types of event logs: (i) real-life logs, i.e., logs of real-life process execution data; (ii) synthetic logs, generated by replaying real-life process models; and (iii) artificial logs, generated by replaying artificial models.

We found that the majority of methods (31 out of 35) were tested using real-life logs. Among them, 11 approaches (cf. [25, 26, 33, 67, 69, 68, 87, 77, 75, 93, 19]) were further tested against synthetic logs, while 13 approaches (cf. [55, 63, 16, 56, 12, 72, 71, 68, 79, 84, 83, 96, 19]) against artificial logs. Finally, one method was tested both on synthetic and artificial logs only (cf. [73]), while [88, 24] were tested on artificial logs and [49] on synthetic logs only. Among the methods that employ real-life logs, we observed a growing trend in employing publicly-available logs, as opposed to private logs which hamper the replicability of the results due to not being accessible.

Concerning the application domains of the real-life logs, we noticed that several methods used a selection of the logs made available by the Business Process Intelligence Challenge (BPIC), which is held annually as part of the BPM Conference series. These logs are publicly available at the 4TU Centre for Research Data, 444https://data.4tu.nl/repository/collection:event_logs_real and cover domains such as healthcare (used by [46, 33, 38, 71, 85, 92, 19]), banking (used by [33, 38, 62, 67, 72, 77, 85, 81, 92, 19, 96]), IT support management in automotive (cf. [75, 77, 92, 19]), and public administration (cf. [26, 38, 55, 16, 96]). A public log pertaining to a process for managing road traffic fines (also available at the 4TU Centre for Research Data) was used in [92, 19]. In [84], the authors use logs from various domains available at http://www.processmining.be/actitrac/.

Besides these publicly-available logs, a range of private logs were also used, originating from different domains such as logistics (cf. [87, 89]), traffic congestion dynamics [69], employers habits (cf. [56, 81]), automotive [12], healthcare [72, 92, 19], and project management and insurance (cf. [79, 47]).

4 Benchmark

Using a selection of the methods surveyed in this paper, we conducted an extensive benchmark to identify relative advantages and tradeoffs. In this section, we justify the criteria of the methods selection, describe the datasets, the evaluation setup and metrics, and present the results of the benchmark. These results, consolidated with the findings from the systematic literature review, are then discussed in Section 5.

4.1 Methods selection

Assessing all the methods that resulted from the search would not be possible due to the heterogeneous nature of the inputs required and the outputs produced. Hence, we decided to focus on the largest subset of comparable methods. The methods considered were the ones satisfying the following criteria:

• i

an implementation of the method is publicly accessible;

• ii

the output of the method is a Petri net or a model seamlessly convertible into a Petri net (i.e., process trees and BPMN models).

The second criterion is a requirement dictated by the metrics used to evaluate the accuracy of a discovered model (illustrated later in this section) that can only be computed on top of Petri nets.

The application of these criteria resulted in an initial selection of the following methods (corresponding to one third of the total studies): $\alpha$$ [12], Inductive Miner [39], Evolutionary Tree Miner [16], Fodina [93], Structured Heuristic Miner 6.0 [19], Split Miner [92], Hybrid ILP Miner [103], RegPFA [77], Stage Miner [95], BPMN Miner [79], Decomposed Process Mining [100].

A posteriori, we excluded the latter four due to the following reasons: Decomposed Process Mining, BPMN Miner, and Stage Miner were excluded as such approaches follow a divide-and-conquer approach which could be applied on top of any discovery method to improve its results; on the other hand, we excluded RegPFA because its output is a graphical representation of a Petri net (DOT), which could not be seamlessly serialized into the standard Petri net format.

We also considered including commercial process mining tools in the benchmark. Specifically, we investigated Disco555http://fluxicon.com/disco, Celonis666http://www.celonis.com, Minit777http://minitlabs.com/, and myInvenio888http://www.my-invenio.com. Disco and Minit are not able to produce business process models from event logs. Instead, they can only produce directly-follows graphs, which do not have an execution semantics. Indeed, when a given node (task) has several incoming arcs, a directly-follows graph does not tell us whether or not the task in question should wait for all its incoming tasks to complete, or just for one of them, or a subset of them. A similar remark applies to split points in the directly-follows graph. Given their lack of execution semantics, it is not possible to directly translate a directly-follows graph into a BPMN models or a Petri net. Instead, one has to determine what is the intended behavior at each split and join point, which is precisely what several of the automated process discovery techniques based on directly-follows graph do (e.g., the Inductive Miner or Split Miner).

Celonis and myInvenio can produce BPMN process models but all they do is to insert OR (inclusive) gateways at the split and join points of the process map. The resulting BPMN process models cannot be translated into Petri nets using existing mappings from BPMN to Petri nets [108]. The Split Miner adopts a similar approach (it initially inserts OR-join gateways at join points in the directly-follows graph), but it then applies an algorithm to turn these OR-join gateways into combinations of XOR and AND gateways – at the expense of potentially making the process model unsafe (but deadlock-free). It would have been possible to take the output of Celonis and myInvenio and apply an approach similar to the Split Miner to remove the OR-join gateways, but the resulting technique would be in essence the Split Miner itself, and the latter is already included in the benchmark.

In conclusion, the final selection of methods for the benchmark contained: $\alpha$$, Inductive Miner (IM), Evolutionary Tree Miner (ETM), Fodina (FO), Structured Heuristic Miner 6.0 (S-HM6), Split Miner (SM), and Hybrid ILP Miner (HILP).

4.2 Setup and datasets

To guarantee the reproducibility of our benchmark and to provide the community with a tool for comparing new methods with the ones evaluated in this paper, we developed a command-line Java application that performs measurements of accuracy and complexity metrics on the seven discovery methods selected above. The tool can be easily extended to incorporate new discovery methods and metrics.999Tool available at https://github.com/raffaeleconforti/ResearchCode/releases/download/v0.1/Benchmark.v0.1.zip and its source code is available at https://github.com/raffaeleconforti/ResearchCode.

For our evaluation, we used two datasets. The first is the collection of real-life event logs publicly available at the 4TU Centre for Research Data as of March 2017.101010https://data.4tu.nl/repository/collection:event_logs_real Out of this collection, we considered the BPI Challenge (BPIC) logs, the Road Traffic Fines Management Process (RTFMP) log, and the SEPSIS Cases log. These logs record executions of business processes from a variety of domains, e.g., healthcare, finance, government and IT service management. For our evaluation we held out those logs that do not explicitly capture business processes (i.e., the BPIC 2011 and 2016 logs), and those contained in other logs (e.g., the Environmental permit application process log). Finally, in three logs (i.e., the BPIC14, BPIC15, and BPIC17 logs), we applied the filtering technique proposed in [109] to remove infrequent behavior. This was necessary since all the models discovered by the considered methods exhibited very poor accuracy (F-score close to 0 or not computable) on these logs, making the comparison useless.

Table III reports the characteristics of the twelve logs used. These logs are widely heterogeneous ranging from simple to very complex, with a log size ranging from 681 traces (for the BPIC152f log) to 150.370 traces (for the RTFMP log). A Similar variety can be observed in the percentage of distinct traces, ranging from 0,2% to 80,6%, and the number of event classes (i.e., activities executed within the process), ranging from 7 to 82. Finally, the length of a trace also varies from very short, with traces containing only one event, to very long with traces containing 185 events.

The second dataset is composed of twelve proprietary logs sourced from several companies around the world. Table IV reports the characteristics of these logs. Also in this case, the logs are quite heterogeneous, with the number of traces (and the percentage of distinct traces) ranging from 225 (of which 99,9% distinct) to 787.657 (of which 0,01% distinct). The number of recorded events varies between 4.434 and 2.099.835, whilst the number of event classes ranges from 8 to 310.

We performed our benchmark on a 6-core Intel Xeon CPU E5-1650 v3 @ 3.50GHz with 128GB RAM running Java 8. We allocated a total of 16GB to the heap space, and we enforced a timeout of four hours for the discovery phase and one hour for measuring each of the quality metrics.

4.3 Evaluation metrics

For all selected discovery metrics we measured the following accuracy and complexity metrics: recall (a.k.a. fitness), precision, generalization, complexity, and soundness.

Fitness measures the ability of a model to reproduce the behavior contained in a log. Under trace semantics, a fitness of 1 means that the model can reproduce every trace in the log. In this paper, we use the fitness measure proposed in [110], which measures the degree to which every trace in the log can be aligned (with a small number of errors) with a trace produced by the model. In other words, this measures tells us how close on average a given trace in the log can be aligned with a trace that can be generated by the model.

Precision measures the ability of a model to generate only the behavior found in the log. A score of 1 indicates that any trace produced by the model is contained in the log. In this paper, we use the precision measure defined in [111], which is based on similar principles as the above fitness measure. Recall and precision can be combined into a single measure known as F-score, which is the harmonic mean of the two measurements $\left(2\cdot\frac{\mathit{Fitness}\cdot\mathit{Precision}}{\mathit{Fitness}+\mathit{Precision}}\right)$.

Generalization refers to the ability of an automated discovery algorithm to discover process models that generate traces that are not present in the log but that can be produced by the business process under observation. In other words, an automated process discovery algorithm has a high generalization on a given event log if it is able to discover a process model from the event log, which generates traces that: (i) are not in the event log, but (ii) can be produced by the business process that produced the event log. Note that unlike fitness and precision, generalization is a property of an algorithm on an event log, and not a property of the model produced by an algorithm when applied to a given event log.

In line with the above definition, we use k-fold cross-validation [112] to measure event logs. This k-fold cross-validation approach to measure generalization has been advocated in several studies in the field of automated process discovery [113, 2, 114, 115]. Concretely, we divide the log into $k$ parts, we discover a model from $k-1$ parts (i.e., we hold-out one part), and measure the fitness of the discovered model against the part held out. This is repeated for every possible part held out. Generalization is the mean of the fitness values obtained for each part held out. A generalization of one means that the discovered model produces traces in the observed process, even if those traces are not in the log from which the model was discovered, and that the discovered model is accurate and does not introduce extra behavior (i.e., does not over-generalize the behavior recorded in the log).

In the results reported below, we use a value of $k=3$ for performance reasons (as opposed to the classical value of $k=10$). The fitness calculation for most of the algorithm-log pairs is slow, and repeating it $10$ times, for every algorithm-log combination is costly. To test if the results could be affected by this choice of $k$, we used $k=10$ for SM and IM on the BPIC12 log, and found that the value of the 10-fold generalization measure was within one percentage point of that of the 3-fold generalization measure.

Complexity quantifies how difficult it is to understand a model. Several complexity metrics have been shown to be (inversely) related to understandability [116], including Size (number of nodes); Control-Flow Complexity (CFC) (the amount of branching caused by gateways in the model) and Structuredness (the percentage of nodes located directly inside a block-structured single-entry single-exit fragment).

Lastly, soundness assesses the behavioral quality of a process model by reporting whether the model violates one of the three soundness criteria [117]: i) option to complete, ii) proper completion, and iii) no dead transitions.

4.4 Benchmark results

We performed two types of evaluations. The first evaluation was meant to compare all the process discovery methods using their default parameters. In the second evaluation, we wanted to perform a similar comparison using hyper-parameter optimization. Due to their extremely long execution times, for this second evaluation we held out $\alpha$$ and ETM which would have been prohibitive for a hyper-parameter optimization exercise. Additionally, we excluded HILP since we did not find any input parameters which could be used to optimize the f-score of the models produced. For the remaining four methods, we evaluated the following input parameters: the two filtering thresholds required in input by SM and S-HM6, the single threshold required in input by IM, and the threshold and the boolean flag required in input by FO. Since all the thresholds range from 0.0 to 1.0, we used steps of 0.05 for IM, and steps of 0.10 for the thresholds of SM, S-HM6 and FO. For FO, we considered all the possible combinations of the filtering threshold and the boolean flag.111111This to have a similar number of data points across all methods.

The results of the first evaluation are shown in the tables V, VI, VII, and VIII, where the best score for each measure on each log is highlighted in bold. In the tables, we used “-” to report that a given accuracy or complexity measurement could not be reliably obtained due to syntactical or behavioral issues in the discovered model (i.e., a disconnected model or an unsound model). Additionally, to report the occurrence of a timeout or an exception during the execution of a discovery method we used “t/o” and “ex”, respectively.

The first evaluation shows the absence of a clear winner among the discovery methods tested, although almost each of them clearly showed specific benefits and drawbacks.

HILP experienced severe difficulties in producing useful outputs. The method often produced disconnected models or models containing multiple end places without providing information about the final marking (a well defined final marking is required in order to measure fitness and precision). Due to these difficulties, we could only assess model complexity for HILP, except for the simplest event log (the PRT5), where HILP had performance comparable to the other methods.

\alpha\$$ showed scalability issues, timing out in eight event logs (33% of the times). Although none of the discovered models stood out in accuracy or in complexity, \alpha$$ in general produced models striking a good balance between fitness and precision (except for the BPIC13inc log).

FO struggled in delivering sound models, discovering only eight sound models. Nevertheless, its outputs were usually highly fitting, scoring five times the best fitness.

S-HM6 performed better than FO, although it also ended up producing unsound models. Of the 16 sound models discovered, nine scored the best fitness. However, precision varied according to the input event log, demonstrating that the performance of this method is bound to the type of log provided in input.

The remaining three methods, i.e., IM, ETM, and SM, consistently performed very well across the whole evaluation, excelling either in fitness, precision or f-score. IM scored 20 times a fitness greater than 0.90 (of which 8 times the highest), though, IM did not stand out for its precision. ETM and SM achieved 19 times a precision greater than 0.80, and ETM precision was the best 10 times. However, ETM scored high precision at the cost of lower fitness. Lastly, SM stood out for its F-score (i.e., high and balanced fitness and precision), achieving an F-score above 0.80 and outperforming the other methods for 18 times. Despite such good results, SM was not able to consistently produces sound models, as showed in the case of the PRT11 log.

In terms of complexity, IM, ETM, and SM delivered good results. IM and ETM always discovered sound and fully block-structured models (struct. of 1.00). ETM and SM discovered models that were within the two smallest models for more than the 80% of the inputs, and that had low CFC, being the ones with the lowest CFC respectively on 14 and 6 logs. On the execution time, SM was the clear winner. It systematically outperformed all the other methods, regardless the input. It was the fastest discovery method 23 times out of 24, discovering a model in less than a second over 19 logs. On the other hand, ETM was the slowest discovery method, reaching the timeout of four hours over 22 event logs.

Finally, Table IX displays the results of the hyper-parameter evaluation. The purpose of this second evaluation was to explore the solutions’ space of each discovery method, to understand if they can achieve higher F-score when optimally tuned. According to this aim, the results of the hyper-parameter evaluation are extremely positive. All the discovery methods were able to improve their F-scores for almost all the inputs (values highlighted in italic in Table IX). Precisely, IM, FO, S-HM6, and SM achieved better F-scores in 19, 14, 15, and 11 event logs, respectively. This denotes that IM default parameters are not optimal when focusing on the discovery of models with high F-score. FO and S-HM6 default parameters are more reliable. Whilst, SM showed to achieve the best with the default parameters more than the 50% of the times. Lastly, SM again outperformed all the other methods, scoring the highest hyper-parameter optimized F-score over 20 event logs.

In conclusion, a method outperforming all other across all metrics could not be identified. All that aside, IM, ETM and SM showed to be the most effective methods when the focus is either on fitness, precision, or F-score, respectively. Nevertheless, all these three methods suffer from one common weakness, which is the inability to handle large-scale real-life logs, as reported for the PRT11 log in our evaluation.

5 Discussion

Our review highlights a growing interest in the field of automated process discovery, and confirms the existence of a wide and heterogeneous number of proposals. Despite such a variety, we can clearly identify two main streams: methods that output procedural process models, and methods that output declarative process models. Further, while the latter ones only rely on declarative statements to represent a process, the former provide various language alternatives, though, most of these methods output Petri nets.

The predominance of Petri nets is driven by the expressive power of this language, and by the requirements of the methods used to assess the quality of the discovered process models (chiefly, fitness and precision). Despite some modeling languages have a straightforward conversion to Petri nets, the strict requirements of these quality assessment tools represent a limitation for the proposals in this research field. For the same reason, it was not possible to compare the two main streams, so we decided to focus our evaluation and comparison on the procedural methods, which in any case, have a higher practical relevance than their declarative counterparts, given that declarative process models are hardly used in practice.

Our benchmark shows benefits and drawbacks of the procedural automated process discovery methods, as well their limitations. These latter include lack of scalability for large and complex logs, and strong differences in the output models, across the various quality metrics. Regarding this aspect, the majority of methods were not able to excel in accuracy or complexity, except for IM, ETM and SM. Indeed, these three ones were the only ones to consistently perform very well in fitness (IM), precision (ETM, SM), F-score (SM), complexity (IM, ETM, SM) and execution time (SM). Nevertheless, our evaluation shows that even IM, ETM and SM can fail when challenged with large-scale unfiltered real-life events logs, as shown in the case of the event log PRT11.

To conclude, even if many proposals are available in this research area, and some of them are able to systematically deliver good to optimal results, there is still space for research and improvements. Furthermore, it is important to highlight that the great majority of the methods do not have a working or available implementation. This hampers their systematic evaluation, so one can only rely on the results reported in the respective papers. Finally, for those methods we assessed, we were not able to identify a unique winner, since the best methods showed to either maximize fitness, precision or F-score. Despite these considerations, it can be noted that there has been significant progress in this field in the past five years. Indeed, IM, ETM and SM clearly outperformed the discovery methods developed in the previous decade and their extensions (i.e., Agnes and S-HM6).

6 Threats to Validity

The first threat to validity refers to the potential selection bias and inaccuracies in data extraction and analysis typical of literature reviews. In order to minimize such issues, our systematic literature review carefully adheres to the guidelines outlined in [4]. Concretely, we used well-known literature sources and libraries in information technology to extract relevant works on the topic of automated process discovery. Further, we performed a backward reference search to avoid the exclusion of potentially relevant papers. Finally, to avoid that our review was threatened by insufficient reliability, we ensured that the search process could be replicated by other researchers. However, the search may produce different results as the algorithm used by source libraries to rank results based on relevance may be updated (see, e.g., Google Scholar).

The experimental evaluation on the other hand is limited in scope to techniques that produce Petri nets (or models in languages such as BPMN or Process Trees, which can be directly translated to Petri nets). Also, it only considers main studies identified in the SLR with an available implementation. In order to compensate for these shortcomings, we published the benchmarking toolset as open-source software in order to enable researchers both to reproduce the results herein reported and to run the same evaluation for other methods, or for alternative configurations of the evaluated methods.

Another limitation is the use of only 24 event logs, which to some extent limits the generalizability of the conclusions. However, the event logs included in the evaluation are all real-life logs of different sizes and features, including different application domains. To mitigate this limitation, we have structured the released benchmarking toolset in such a way that the benchmark can be seamlessly rerun with additional datasets.

7 Related Work

A previous survey and benchmark of automated process discovery methods has been reported by De Weerdt et al. [3]. This survey covered 27 approaches, and it assessed 7 of them. We used it as starting point for our study.

The benchmark reported by De Weerdt et al. [3] includes seven approaches, namely AGNEsMiner, $\alpha{+}$, $\alpha{++}$, Genetic Miner (and a variant thereof), Heuristics Miner and ILP Miner. In comparison, our benchmark includes \alpha$$ (which is an improved version of \alpha{+}$and$\alpha{++}$), Structured Heuristics miner (which is an extension of Heuristics Miner), Hybrid ILP Miner (an improvement of ILP), Evolutionary Tree Miner (which is a genetic algorithm postdating the evaluation of De Weerdt et al. [3]) Notably, we did not include AGNEsMiner due to the very long execution times (as suggested by the authors in a conversation over emails exchanged during this work).

Another difference with respect to the previous survey [3], is that in our paper we based our evaluation both on public and proprietary event logs, whilst the evaluation of De Weerdt et al. [3] is solely based on artificial event logs and closed datasets, due to the unavailability of public datasets at the time of that study.

In terms of results, De Weerdt et al.[3] found that Heuristics Miner achieved a better F-score than other approaches and generally produced simpler models, while ILP achieved the best fitness at the expense of low precision and high model complexity. Our results show that SM achieves even better F-score and lower model complexity than other techniques, followed by ETM and IM, which excelled for precision and fitness (respectively). Thus it appears that in this field, in the last years, progress has been pursued successfully.

Another previous survey in the field is outdated [118] and a more recent one is not intended to be comprehensive [119], but rather limits on plug-ins available in the ProM toolset. Another related effort is CoBeFra – a tool suite for measuring fitness, precision and model complexity of automatically discovered process models [120].

8 Conclusion

This article presented a Systematic Literature Review (SLR) of automated process discovery methods and a comparative evaluation of existing implementations of these methods using a benchmark covering twelve publicly-available real-life event logs, twelve proprietary real-life event logs, and nine quality metrics. The toolset used in this benchmark is available as open-source software and the 50% of the event logs are publicly available. The benchmarking toolset has been designed in a way that it can be seamlessly extended with additional methods, event logs, and evaluation metrics.

The SLR put into evidence a vast number of automated process discovery methods (344 relevant papers were analysed). Traditionally, many of these proposals produce Petri nets, but more recently, we observe an increasing number of methods that produce models in other languages, including BPMN and declarative constraints. We also observe a recent emphasis on methods that produce block-structured process models.

The results of the empirical evaluation show that methods that seek to produce block-structured process models (Inductive Miner and Evolutionary Tree Miner) achieve the best performance in terms of fitness or precision, and complexity. Whilst, methods that do not restrict the topology of the generated process models (Split Miner), produce process models of higher quality in terms of F-score, although these methods cannot guarantee soundness (though they can guarantee deadlock-freedom). We also observed that in the case of very complex event logs, it is necessary to use a filtering method prior to applying existing automated process discovery methods. Without this filtering, the precision of the resulting models was close to zero. A direction for future work is to develop automated process discovery techniques that incorporate adaptive filtering approaches so that they can auto-tune themselves to deal with very complex logs.

Another limitation observed while conducting the benchmark, was the lack of universal measures of fitness and precision, which would be applicable not only to Petri nets (or BPMN models that can be mapped to Petri nets), but equally well to declarative or data-driven process modeling notations. Developing more universal measures of fitness and precision is another possible target of future work.

Acknowledgments

This research is partly funded by the Australian Research Council (grant DP150103356) and the Estonian Research Council (grant IUT20-55). It is also partly supported by the H2020-RISE EU project FIRST (734599), the Sapienza grant DAKIP and the Italian projects Social Museum and Smart Tourism (CTN01_00034_23154), NEPTIS (PON03PE_00214_3), and RoMA - Resilence of Metropolitan Areas (SCN_00064).