Assisted Data Annotation for Business Process Information Extraction from Textual Documents

TL;DR

This paper presents assistance features, including recommendations and visualization, to improve the creation of high-quality datasets for business process information extraction from text, reducing workload and enhancing annotation quality.

Contribution

It introduces novel assistance features for dataset creation in process information extraction, supported by empirical evidence of workload reduction and quality improvement.

Findings

Assistance features reduce annotation workload by up to 51%.

Recommendations significantly improve annotation quality by up to 38.9%.

All data and code are publicly available for further research.

Abstract

Machine-learning based generation of process models from natural language text process descriptions provides a solution for the time-intensive and expensive process discovery phase. Many organizations have to carry out this phase, before they can utilize business process management and its benefits. Yet, research towards this is severely restrained by an apparent lack of large and high-quality datasets. This lack of data can be attributed to, among other things, an absence of proper tool assistance for dataset creation, resulting in high workloads and inferior data quality. We explore two assistance features to support dataset creation, a recommendation system for identifying process information in the text and visualization of the current state of already identified process information as a graphical business process model. A controlled user study with 31 participants shows that assisting dataset creators with recommendations lowers all aspects of workload, up to $-51.0\%$, and significantly improves annotation quality, up to $+38.9\%$. We make all data and code available to encourage further research on additional novel assistance strategies.