FABLE: A Novel Data-Flow Analysis Benchmark on Procedural Text for Large Language Model Evaluation

TL;DR

FABLE is a comprehensive benchmark that evaluates large language models' ability to understand data flow in procedural texts across multiple domains, revealing current limitations and guiding future improvements.

Contribution

Introduces FABLE, the first systematic benchmark for assessing LLMs' data-flow reasoning in procedural contexts across diverse real-world domains.

Findings

Reasoning-focused models outperform general-purpose models in accuracy.

Current LLMs perform near random chance on data-flow reasoning tasks.

FABLE enables diagnostic evaluation of procedural understanding in LLMs.

Abstract

Understanding how data moves, transforms, and persists, known as data flow, is fundamental to reasoning in procedural tasks. Despite their fluency in natural and programming languages, large language models (LLMs), although increasingly being applied to decisions with procedural tasks, have not been systematically evaluated for their ability to perform data-flow reasoning. We introduce FABLE, an extensible benchmark designed to assess LLMs' understanding of data flow using structured, procedural text. FABLE adapts eight classical data-flow analyses from software engineering: reaching definitions, very busy expressions, available expressions, live variable analysis, interval analysis, type-state analysis, taint analysis, and concurrency analysis. These analyses are instantiated across three real-world domains: cooking recipes, travel routes, and automated plans. The benchmark includes 2,400 question-answer pairs, with 100 examples for each domain-analysis combination. We evaluate three types of LLMs: a reasoning-focused model (DeepSeek-R1 8B), a general-purpose model (LLaMA 3.1 8B), and a code-specific model (Granite Code 8B). Each model is tested using majority voting over five sampled completions per prompt. Results show that the reasoning model achieves higher accuracy, but at the cost of over 20 times slower inference compared to the other models. In contrast, the general-purpose and code-specific models perform close to random chance. FABLE provides the first diagnostic benchmark to systematically evaluate data-flow reasoning and offers insights for developing models with stronger procedural understanding.