Accurate and Regret-aware Numerical Problem Solver for Tabular Question Answering

TL;DR

This paper introduces TabLaP, a novel table question answering model that combines LLMs for multi-step reasoning with a Python interpreter for precise numerical calculations, and assesses answer trustworthiness.

Contribution

It proposes a new approach using LLMs as planners and a Python interpreter for calculations, improving accuracy and enabling regret-aware answer confidence estimation.

Findings

TabLaP outperforms state-of-the-art models with 5.7% and 5.8% accuracy improvements.

The model effectively combines reasoning and precise numerical computation.

Trustworthiness estimation allows regret-aware decision making.

Abstract

Question answering on free-form tables (a.k.a. TableQA) is a challenging task because of the flexible structure and complex schema of tables. Recent studies use Large Language Models (LLMs) for this task, exploiting their capability in understanding the questions and tabular data, which are typically given in natural language and contain many textual fields, respectively. While this approach has shown promising results, it overlooks the challenges brought by numerical values which are common in tabular data, and LLMs are known to struggle with such values. We aim to address this issue, and we propose a model named TabLaP that uses LLMs as a planner rather than an answer generator. This approach exploits LLMs' capability in multi-step reasoning while leaving the actual numerical calculations to a Python interpreter for accurate calculation. Recognizing the inaccurate nature of LLMs, we further make a first attempt to quantify the trustworthiness of the answers produced by TabLaP, such that users can use TabLaP in a regret-aware manner. Experimental results on two benchmark datasets show that TabLaP is substantially more accurate than the state-of-the-art models, improving the answer accuracy by 5.7% and 5.8% on the two datasets, respectively.