StrucText-Eval: Evaluating Large Language Model's Reasoning Ability in Structure-Rich Text

TL;DR

This paper introduces StrucText-Eval, a benchmark for evaluating large language models' reasoning abilities on structure-rich text, revealing current limitations in understanding complex structured data.

Contribution

It presents a novel automatic data generation method and a comprehensive benchmark supporting multiple languages and tasks to assess LLM reasoning on structured data.

Findings

Open-source LLMs achieve up to 74.9% accuracy on standard tasks.

Performance drops to 45.8% on harder tasks.

Humans reach 92.6% accuracy on complex structured data.

Abstract

The effective utilization of structured data, integral to corporate data strategies, has been challenged by the rise of large language models (LLMs) capable of processing unstructured information. This shift prompts the question: can LLMs interpret structured data directly in its unstructured form? We propose an automatic evaluation data generation method for assessing LLMs' reasoning capabilities on structure-rich text to explore this. Our approach supports 8 structured languages and 29 tasks, generating data with adjustable complexity through controllable nesting and structural width. We introduce StrucText-Eval, a benchmark containing 5,800 pre-generated and annotated samples designed to evaluate how well LLMs understand and reason through structured text. StrucText-Eval is divided into two suites: a regular Test suite (3,712 samples) and a Test-Hard suite (2,088 samples), the latter emphasizing the gap between human and model performance on more complex tasks. Experimental results show that while open-source LLMs achieve a maximum accuracy of 74.9\% on the standard dataset, their performance drops significantly to 45.8\% on the harder dataset. In contrast, human participants reach an accuracy of 92.6\% on StrucText-Eval-Hard, highlighting LLMs' current limitations in handling intricate structural information. The benchmark and generation codes are open sourced in \url{https://github.com/MikeGu721/StrucText-Eval}