Benchmarking Table Comprehension In The Wild
Yikang Pan, Yi Zhu, Rand Xie, Yizhi Liu

TL;DR
This paper introduces TableQuest, a comprehensive benchmark for evaluating large language models' ability to understand and reason with tables in real-world financial reports, highlighting current limitations and failure modes.
Contribution
The paper presents a new holistic benchmark, TableQuest, for assessing LLMs' table comprehension in realistic contexts, addressing gaps in prior isolated and narrow skill benchmarks.
Findings
Models perform well in fact retrieval but struggle with complex reasoning.
Multi-step calculations remain challenging for current models.
Benchmark and evaluation procedures are publicly available.
Abstract
Large Language Models (LLMs), while being increasingly dominant on a myriad of knowledge-intensive activities, have only had limited success understanding lengthy table-text mixtures, such as academic papers and financial reports. Recent advances of long-context LLMs have opened up new possibilities for this field. Nonetheless, we identify two roadblocks: (1) Prior benchmarks of table question answering (TableQA) have focused on isolated tables without context, making it hard to evaluate models in real-world scenarios. (2) Prior benchmarks have focused on some narrow skill sets of table comprehension such as table recognition, data manipulation/calculation, table summarization etc., while a skilled human employs those skills collectively. In this work, we introduce TableQuest, a new benchmark designed to evaluate the holistic table comprehension capabilities of LLMs in the natural…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
