Detecting Undesired Process Behavior by Means of Retrieval Augmented Generation

TL;DR

This paper introduces a retrieval augmented generation approach that enables detection of undesired process behaviors without the need for process models or resource-intensive fine-tuning of language models, outperforming traditional methods.

Contribution

The authors propose a novel RAG-based method that leverages a knowledge base to detect undesired process behavior without fine-tuning, addressing resource and generalization issues.

Findings

RAG outperforms fine-tuned LLMs in detecting undesired behavior.

The approach benefits from context like frequent traces and activities.

It eliminates the need for dedicated process models or fine-tuning.

Abstract

Conformance checking techniques detect undesired process behavior by comparing process executions that are recorded in event logs to desired behavior that is captured in a dedicated process model. If such models are not available, conformance checking techniques are not applicable, but organizations might still be interested in detecting undesired behavior in their processes. To enable this, existing approaches use Large Language Models (LLMs), assuming that they can learn to distinguish desired from undesired behavior through fine-tuning. However, fine-tuning is highly resource-intensive and the fine-tuned LLMs often do not generalize well. To address these limitations, we propose an approach that requires neither a dedicated process model nor resource-intensive fine-tuning to detect undesired process behavior. Instead, we use Retrieval Augmented Generation (RAG) to provide an LLM with direct access to a knowledge base that contains both desired and undesired process behavior from other processes, assuming that the LLM can transfer this knowledge to the process at hand. Our evaluation shows that our approach outperforms fine-tuned LLMs in detecting undesired behavior, demonstrating that RAG is a viable alternative to resource-intensive fine-tuning, particularly when enriched with relevant context from the event log, such as frequent traces and activities.