RealHiTBench: A Comprehensive Realistic Hierarchical Table Benchmark for Evaluating LLM-Based Table Analysis

TL;DR

RealHiTBench is a new comprehensive benchmark that challenges LLMs and MLLMs in understanding complex, hierarchical, and multi-format tabular data, promoting advancements in table reasoning capabilities.

Contribution

The paper introduces RealHiTBench, a diverse and challenging benchmark for evaluating LLMs on complex tabular data, and proposes TreeThinker, a hierarchical reasoning pipeline to improve table understanding.

Findings

RealHiTBench is more challenging than existing benchmarks.

State-of-the-art LLMs show limited performance on RealHiTBench.

TreeThinker enhances LLMs' reasoning over hierarchical table structures.

Abstract

With the rapid advancement of Large Language Models (LLMs), there is an increasing need for challenging benchmarks to evaluate their capabilities in handling complex tabular data. However, existing benchmarks are either based on outdated data setups or focus solely on simple, flat table structures. In this paper, we introduce RealHiTBench, a comprehensive benchmark designed to evaluate the performance of both LLMs and Multimodal LLMs (MLLMs) across a variety of input formats for complex tabular data, including LaTeX, HTML, and PNG. RealHiTBench also includes a diverse collection of tables with intricate structures, spanning a wide range of task types. Our experimental results, using 25 state-of-the-art LLMs, demonstrate that RealHiTBench is indeed a challenging benchmark. Moreover, we also develop TreeThinker, a tree-based pipeline that organizes hierarchical headers into a tree structure for enhanced tabular reasoning, validating the importance of improving LLMs' perception of table hierarchies. We hope that our work will inspire further research on tabular data reasoning and the development of more robust models. The code and data are available at https://github.com/cspzyy/RealHiTBench.