Understanding Unnatural Questions Improves Reasoning over Text

TL;DR

This paper introduces a method to improve complex question answering by converting natural questions into unnatural ones that are easier to parse, enabling training with synthetic data and outperforming models trained on human-labeled data.

Contribution

It proposes a novel projection model that maps natural questions to unnatural questions for better parsing, reducing reliance on human-annotated data.

Findings

Synthetic training data outperforms human-labeled data in QA tasks.

Projection model effectively maps natural questions to unnatural ones.

Method reduces data annotation costs for complex question answering.

Abstract

Complex question answering (CQA) over raw text is a challenging task. A prominent approach to this task is based on the programmer-interpreter framework, where the programmer maps the question into a sequence of reasoning actions which is then executed on the raw text by the interpreter. Learning an effective CQA model requires large amounts of human-annotated data,consisting of the ground-truth sequence of reasoning actions, which is time-consuming and expensive to collect at scale. In this paper, we address the challenge of learning a high-quality programmer (parser) by projecting natural human-generated questions into unnatural machine-generated questions which are more convenient to parse. We firstly generate synthetic (question,action sequence) pairs by a data generator, and train a semantic parser that associates synthetic questions with their corresponding action sequences. To capture the diversity when applied tonatural questions, we learn a projection model to map natural questions into their most similar unnatural questions for which the parser can work well. Without any natural training data, our projection model provides high-quality action sequences for the CQA task. Experimental results show that the QA model trained exclusively with synthetic data generated by our method outperforms its state-of-the-art counterpart trained on human-labeled data.