Business as Rulesual: A Benchmark and Framework for Business Rule Flow Modeling with LLMs

TL;DR

This paper introduces BREX, a comprehensive benchmark dataset of real-world business documents, and ExIde, a reasoning framework leveraging large language models to extract and model complex business rules with logical dependencies.

Contribution

The paper presents BREX, a large-scale, multi-domain benchmark for business rule extraction, and ExIde, a novel reasoning framework that improves rule dependency modeling without fine-tuning.

Findings

Executable grounding outperforms standard prompts in rule extraction.

Reasoning-optimized models excel at tracing complex rule dependencies.

ExIde achieves significant improvements across diverse LLMs.

Abstract

Extracting structured procedural knowledge from unstructured business documents is a critical yet unresolved bottleneck in process automation. While prior work has focused on extracting linear action flows from instructional texts, such as recipes, it has insufficiently addressed the complex logical structures, including conditional branching and parallel execution, that are pervasive in real-world regulatory and administrative documents. Furthermore, existing benchmarks are limited by simplistic schemas and shallow logical dependencies, restricting progress toward logic-aware large language models.To bridge this Logic Gap, we introduce BREX, a carefully curated benchmark comprising 409 real-world business documents and 2,855 expert-annotated rules. Unlike prior datasets centered on narrow service scenarios, BREX spans over 30 vertical domains, covering scientific, industrial, administrative, and financial regulations. We further propose ExIde, a structure-aware reasoning framework that investigates five distinct prompting strategies, ranging from implicit semantic alignment to executable grounding via pseudo-code generation. This enables explicit modeling of rule dependencies and provides an out-of-the-box framework for different business customers without finetuning their own large language models. We benchmark ExIde using 13 state-of-the-art large language models. Our extensive evaluation reveals that executable grounding serves as a superior inductive bias, significantly outperforming standard prompts in rule extraction. In addition, reasoning-optimized models demonstrate a distinct advantage in tracing long-range and non-linear rule dependencies compared to standard instruction-tuned models.