Generate, Transform, Answer: Question Specific Tool Synthesis for Tabular Data

TL;DR

This paper introduces ToolWriter, a system that generates query-specific programs to transform tables for improved neural question answering, especially on large tables, by mimicking human-like data filtering techniques.

Contribution

We propose ToolWriter, a novel approach that generates and applies data transformation programs to enhance neural TQA models' performance on large, complex tables.

Findings

Improved state-of-the-art results on WikiTableQuestions and WikiSQL.

Significant performance gains on long tables.

Demonstrated potential of programmatic tools in structured data manipulation.

Abstract

Tabular question answering (TQA) presents a challenging setting for neural systems by requiring joint reasoning of natural language with large amounts of semi-structured data. Unlike humans who use programmatic tools like filters to transform data before processing, language models in TQA process tables directly, resulting in information loss as table size increases. In this paper we propose ToolWriter to generate query specific programs and detect when to apply them to transform tables and align them with the TQA model's capabilities. Focusing ToolWriter to generate row-filtering tools improves the state-of-the-art for WikiTableQuestions and WikiSQL with the most performance gained on long tables. By investigating headroom, our work highlights the broader potential for programmatic tools combined with neural components to manipulate large amounts of structured data.