RUST-BENCH: Benchmarking LLM Reasoning on Unstructured Text within Structured Tables

TL;DR

RUST-BENCH is a comprehensive benchmark with nearly 8,000 questions from real-world tables designed to evaluate large language models' reasoning abilities across complex, heterogeneous, and domain-specific data, highlighting current limitations.

Contribution

This work introduces RUST-BENCH, a large-scale, real-world table reasoning benchmark that challenges LLMs with heterogeneity, scale, and multi-hop inference, filling a gap in existing benchmarks.

Findings

LLMs struggle with heterogeneous schemas and multi-hop reasoning

Current models show weaknesses in complex, real-world table reasoning

RUST-BENCH provides a new challenging testbed for future research

Abstract

Existing tabular reasoning benchmarks mostly test models on small, uniform tables, underrepresenting the complexity of real-world data and giving an incomplete view of Large Language Models' (LLMs) reasoning abilities. Real tables are long, heterogeneous, and domain-specific, mixing structured fields with free text and requiring multi-hop reasoning across thousands of tokens. To address this gap, we introduce RUST-BENCH, a benchmark of 7966 questions from 2031 real-world tables spanning two domains: i) RB-Science (NSF grant records) and ii) RB-Sports (NBA statistics). Unlike prior work, RUST-BENCH evaluates LLMs jointly across scale, heterogeneity, domain specificity, and reasoning complexity. Experiments with open-source and proprietary models show that LLMs struggle with heterogeneous schemas and complex multi-hop inference, revealing persistent weaknesses in current architectures and prompting strategies. RUST-BENCH establishes a challenging new testbed for advancing tabular reasoning research.