TReB: A Comprehensive Benchmark for Evaluating Table Reasoning Capabilities of Large Language Models

TL;DR

This paper introduces TReB, a comprehensive benchmark with 26 sub-tasks for evaluating large language models' ability to reason with table-structured data, addressing a critical gap in performance assessment.

Contribution

The paper presents a new benchmark and evaluation framework for table reasoning, including a high-quality dataset and testing across 20+ LLMs, highlighting the need for improved reasoning capabilities.

Findings

Existing LLMs show significant room for improvement in table reasoning.

The benchmark effectively differentiates LLM performance on various table reasoning tasks.

The dataset and framework are publicly available for further research.

Abstract

The majority of data in businesses and industries is stored in tables, databases, and data warehouses. Reasoning with table-structured data poses significant challenges for large language models (LLMs) due to its hidden semantics, inherent complexity, and structured nature. One of these challenges is lacking an effective evaluation benchmark fairly reflecting the performances of LLMs on broad table reasoning abilities. In this paper, we fill in this gap, presenting a comprehensive table reasoning evolution benchmark, TReB, which measures both shallow table understanding abilities and deep table reasoning abilities, a total of 26 sub-tasks. We construct a high quality dataset through an iterative data processing procedure. We create an evaluation framework to robustly measure table reasoning capabilities with three distinct inference modes, TCoT, PoT and ICoT. Further, we benchmark over 20 state-of-the-art LLMs using this frame work and prove its effectiveness. Experimental results reveal that existing LLMs still have significant room for improvement in addressing the complex and real world Table related tasks. Both the dataset and evaluation framework are publicly available, with the dataset hosted on huggingface.co/datasets/JT-LM/JIUTIAN-TReB and the framework on github.com/JT-LM/jiutian-treb.