Benchmarking Table Comprehension In The Wild

TL;DR

This paper introduces TableQuest, a comprehensive benchmark for evaluating large language models' ability to understand and reason with tables in real-world financial reports, highlighting current limitations and failure modes.

Contribution

The paper presents a new holistic benchmark, TableQuest, for assessing LLMs' table comprehension in realistic contexts, addressing gaps in prior isolated and narrow skill benchmarks.

Findings

Models perform well in fact retrieval but struggle with complex reasoning.

Multi-step calculations remain challenging for current models.

Benchmark and evaluation procedures are publicly available.

Abstract

Large Language Models (LLMs), while being increasingly dominant on a myriad of knowledge-intensive activities, have only had limited success understanding lengthy table-text mixtures, such as academic papers and financial reports. Recent advances of long-context LLMs have opened up new possibilities for this field. Nonetheless, we identify two roadblocks: (1) Prior benchmarks of table question answering (TableQA) have focused on isolated tables without context, making it hard to evaluate models in real-world scenarios. (2) Prior benchmarks have focused on some narrow skill sets of table comprehension such as table recognition, data manipulation/calculation, table summarization etc., while a skilled human employs those skills collectively. In this work, we introduce TableQuest, a new benchmark designed to evaluate the holistic table comprehension capabilities of LLMs in the natural table-rich context of financial reports. We employ a rigorous data processing and filtering procedure to ensure that the question-answer pairs are logical, reasonable, and diverse. We experiment with 7 state-of-the-art models, and find that despite reasonable accuracy in locating facts, they often falter when required to execute more sophisticated reasoning or multi-step calculations. We conclude with a qualitative study of the failure modes and discuss the challenges of constructing a challenging benchmark. We make the evaluation data, judging procedure and results of this study publicly available to facilitate research in this field.