Exploring Performance Contrasts in TableQA: Step-by-Step Reasoning Boosts Bigger Language Models, Limits Smaller Language Models

TL;DR

This paper introduces Table-Logic, a step-by-step reasoning prompting flow for TableQA, revealing performance disparities between large and small language models and analyzing the underlying causes.

Contribution

The paper presents a novel prompting method, Table-Logic, to systematically compare and analyze the performance of different-sized language models in TableQA tasks.

Findings

Bigger LMs like Llama-3-70B improve accuracy by 7.8% with Table-Logic.

Smaller LMs like Llama-2-7B decline by 11% in performance using the method.

Performance gaps are linked to the limitations of step-by-step reasoning in small models.

Abstract

This paper proposes a detailed prompting flow, termed Table-Logic, to investigate the performance contrasts between bigger and smaller language models (LMs) utilizing step-by-step reasoning methods in the TableQA task. The method processes tasks by sequentially identifying critical columns and rows given question and table with its structure, determining necessary aggregations, calculations, or comparisons, and finally inferring the results to generate a precise prediction. By deploying this method, we observe a 7.8% accuracy improvement in bigger LMs like Llama-3-70B compared to the vanilla on HybridQA, while smaller LMs like Llama-2-7B shows an 11% performance decline. We empirically investigate the potential causes of performance contrasts by exploring the capabilities of bigger and smaller LMs from various dimensions in TableQA task. Our findings highlight the limitations of the step-by-step reasoning method in small models and provide potential insights for making improvements.