Text2Analysis: A Benchmark of Table Question Answering with Advanced Data Analysis and Unclear Queries

TL;DR

This paper introduces Text2Analysis, a comprehensive benchmark for advanced table question answering that includes complex analysis tasks and unclear queries, highlighting current model limitations and fostering future research.

Contribution

The paper presents a new benchmark with advanced analysis tasks, innovative annotation methods, and real-world-like unclear queries to challenge and evaluate large language models in tabular data analysis.

Findings

Current models struggle with advanced analysis tasks.

Benchmark introduces significant challenges for state-of-the-art models.

New dataset includes 2249 query-result pairs across 347 tables.

Abstract

Tabular data analysis is crucial in various fields, and large language models show promise in this area. However, current research mostly focuses on rudimentary tasks like Text2SQL and TableQA, neglecting advanced analysis like forecasting and chart generation. To address this gap, we developed the Text2Analysis benchmark, incorporating advanced analysis tasks that go beyond the SQL-compatible operations and require more in-depth analysis. We also develop five innovative and effective annotation methods, harnessing the capabilities of large language models to enhance data quality and quantity. Additionally, we include unclear queries that resemble real-world user questions to test how well models can understand and tackle such challenges. Finally, we collect 2249 query-result pairs with 347 tables. We evaluate five state-of-the-art models using three different metrics and the results show that our benchmark presents introduces considerable challenge in the field of tabular data analysis, paving the way for more advanced research opportunities.