What is the Best Process Model Representation? A Comparative Analysis for Process Modeling with Large Language Models

TL;DR

This paper systematically compares various process model representations for process modeling with large language models, introducing a new dataset and evaluating their suitability and performance.

Contribution

It provides the first empirical comparison of multiple process model representations for LLM-based process modeling, including a new dataset and evaluation framework.

Findings

Mermaid achieves the highest overall score in PMo criteria.

BPMN text delivers the best process element similarity in PMG.

The study highlights the strengths and weaknesses of different PMRs.

Abstract

Large Language Models (LLMs) are increasingly applied for Process Modeling (PMo) tasks such as Process Model Generation (PMG). To support these tasks, researchers have introduced a variety of Process Model Representations (PMRs) that serve as model abstractions or generation targets. However, these PMRs differ widely in structure, complexity, and usability, and have never been systematically compared. Moreover, recent PMG approaches rely on distinct evaluation strategies and generation techniques, making comparison difficult. This paper presents the first empirical study that evaluates multiple PMRs in the context of PMo with LLMs. We introduce the PMo Dataset, a new dataset containing 55 process descriptions paired with models in nine different PMRs. We evaluate PMRs along two dimensions: suitability for LLM-based PMo and performance on PMG. \textit{Mermaid} achieves the highest overall score across six PMo criteria, whereas \textit{BPMN text} delivers the best PMG results in terms of process element similarity.