Evaluating the Ability of LLMs to Solve Semantics-Aware Process Mining Tasks

TL;DR

This paper assesses the capabilities of large language models in semantics-aware process mining tasks, revealing they require fine-tuning to outperform smaller models and are ineffective with minimal in-context learning.

Contribution

It provides a systematic evaluation of LLMs for process mining, introduces benchmark datasets, and compares out-of-the-box versus fine-tuned performance.

Findings

LLMs struggle with challenging tasks without fine-tuning

Fine-tuning significantly improves LLM performance

Fine-tuned LLMs outperform smaller encoder-based models

Abstract

The process mining community has recently recognized the potential of large language models (LLMs) for tackling various process mining tasks. Initial studies report the capability of LLMs to support process analysis and even, to some extent, that they are able to reason about how processes work. This latter property suggests that LLMs could also be used to tackle process mining tasks that benefit from an understanding of process behavior. Examples of such tasks include (semantic) anomaly detection and next activity prediction, which both involve considerations of the meaning of activities and their inter-relations. In this paper, we investigate the capabilities of LLMs to tackle such semantics-aware process mining tasks. Furthermore, whereas most works on the intersection of LLMs and process mining only focus on testing these models out of the box, we provide a more principled investigation of the utility of LLMs for process mining, including their ability to obtain process mining knowledge post-hoc by means of in-context learning and supervised fine-tuning. Concretely, we define three process mining tasks that benefit from an understanding of process semantics and provide extensive benchmarking datasets for each of them. Our evaluation experiments reveal that (1) LLMs fail to solve challenging process mining tasks out of the box and when provided only a handful of in-context examples, (2) but they yield strong performance when fine-tuned for these tasks, consistently surpassing smaller, encoder-based language models.