RobuT: A Systematic Study of Table QA Robustness Against Human-Annotated Adversarial Perturbations

TL;DR

This paper introduces RobuT, a benchmark for evaluating the robustness of Table QA models against human-annotated adversarial perturbations, revealing their vulnerability and proposing adversarial training with large language models to improve robustness.

Contribution

The paper presents RobuT, a new benchmark with adversarial perturbations for Table QA, and demonstrates how adversarial training with large language models enhances model robustness.

Findings

State-of-the-art Table QA models are vulnerable to adversarial perturbations.

Large language models can generate effective adversarial examples.

Adversarial training significantly improves robustness of Table QA models.

Abstract

Despite significant progress having been made in question answering on tabular data (Table QA), it's unclear whether, and to what extent existing Table QA models are robust to task-specific perturbations, e.g., replacing key question entities or shuffling table columns. To systematically study the robustness of Table QA models, we propose a benchmark called RobuT, which builds upon existing Table QA datasets (WTQ, WikiSQL-Weak, and SQA) and includes human-annotated adversarial perturbations in terms of table header, table content, and question. Our results indicate that both state-of-the-art Table QA models and large language models (e.g., GPT-3) with few-shot learning falter in these adversarial sets. We propose to address this problem by using large language models to generate adversarial examples to enhance training, which significantly improves the robustness of Table QA models. Our data and code is publicly available at https://github.com/yilunzhao/RobuT.