ExOAR: Expert-Guided Object and Activity Recognition from Textual Data

TL;DR

ExOAR is an interactive method that combines large language models with human verification to extract structured object and activity data from unstructured text, aiding process mining.

Contribution

It introduces a novel, human-in-the-loop approach that leverages LLMs for extracting structured data from unstructured textual sources for process analysis.

Findings

Effective in extracting structured data from unstructured text

Maintains high accuracy with human oversight

Validated on real-world user data

Abstract

Object-centric process mining requires structured data, but extracting it from unstructured text remains a challenge. We introduce ExOAR (Expert-Guided Object and Activity Recognition), an interactive method that combines large language models (LLMs) with human verification to identify objects and activities from textual data. ExOAR guides users through consecutive stages in which an LLM generates candidate object types, activities, and object instances based on contextual input, such as a user's profession, and textual data. Users review and refine these suggestions before proceeding to the next stage. Implemented as a practical tool, ExOAR is initially validated through a demonstration and then evaluated with real-world Active Window Tracking data from five users. Our results show that ExOAR can effectively bridge the gap between unstructured textual data and the structured log with clear semantics needed for object-centric process analysis, while it maintains flexibility and human oversight.