OmniTab: Pretraining with Natural and Synthetic Data for Few-shot Table-based Question Answering

TL;DR

OmniTab introduces a pretraining method using both natural and synthetic data to improve few-shot and full data table-based question answering, achieving state-of-the-art results with minimal annotation.

Contribution

The paper presents a novel omnivorous pretraining approach that combines natural and synthetic data to enhance table-based QA models with minimal annotation effort.

Findings

Achieves 16.2% absolute gain in 128-shot setting

Establishes new state-of-the-art on WikiTableQuestions

Demonstrates effectiveness of combined natural and synthetic pretraining

Abstract

The information in tables can be an important complement to text, making table-based question answering (QA) systems of great value. The intrinsic complexity of handling tables often adds an extra burden to both model design and data annotation. In this paper, we aim to develop a simple table-based QA model with minimal annotation effort. Motivated by the fact that table-based QA requires both alignment between questions and tables and the ability to perform complicated reasoning over multiple table elements, we propose an omnivorous pretraining approach that consumes both natural and synthetic data to endow models with these respective abilities. Specifically, given freely available tables, we leverage retrieval to pair them with relevant natural sentences for mask-based pretraining, and synthesize NL questions by converting SQL sampled from tables for pretraining with a QA loss. We perform extensive experiments in both few-shot and full settings, and the results clearly demonstrate the superiority of our model OmniTab, with the best multitasking approach achieving an absolute gain of 16.2% and 2.7% in 128-shot and full settings respectively, also establishing a new state-of-the-art on WikiTableQuestions. Detailed ablations and analyses reveal different characteristics of natural and synthetic data, shedding light on future directions in omnivorous pretraining. Code, pretraining data, and pretrained models are available at https://github.com/jzbjyb/OmniTab.