Topic Transferable Table Question Answering

TL;DR

This paper introduces T3QA, a framework to improve table question-answering models under topic shift scenarios, by augmenting training data and refining logical form ranking, addressing real-world deployment challenges.

Contribution

The paper presents novel benchmarks for topic shift in TableQA and proposes T3QA, a comprehensive adaptation framework including vocabulary injection, topic-specific question generation, and reranking.

Findings

Models degrade on unseen topics without adaptation.

T3QA improves performance on topic shift benchmarks.

The proposed methods offer a practical baseline for robust TableQA.

Abstract

Weakly-supervised table question-answering(TableQA) models have achieved state-of-art performance by using pre-trained BERT transformer to jointly encoding a question and a table to produce structured query for the question. However, in practical settings TableQA systems are deployed over table corpora having topic and word distributions quite distinct from BERT's pretraining corpus. In this work we simulate the practical topic shift scenario by designing novel challenge benchmarks WikiSQL-TS and WikiTQ-TS, consisting of train-dev-test splits in five distinct topic groups, based on the popular WikiSQL and WikiTableQuestions datasets. We empirically show that, despite pre-training on large open-domain text, performance of models degrades significantly when they are evaluated on unseen topics. In response, we propose T3QA (Topic Transferable Table Question Answering) a pragmatic adaptation framework for TableQA comprising of: (1) topic-specific vocabulary injection into BERT, (2) a novel text-to-text transformer generator (such as T5, GPT2) based natural language question generation pipeline focused on generating topic specific training data, and (3) a logical form reranker. We show that T3QA provides a reasonably good baseline for our topic shift benchmarks. We believe our topic split benchmarks will lead to robust TableQA solutions that are better suited for practical deployment.