e-QRAQ: A Multi-turn Reasoning Dataset and Simulator with Explanations

TL;DR

This paper introduces e-QRAQ, a dataset and simulator for multi-turn reasoning with explanations, enabling agents to read ambiguous stories, ask clarifying questions, and justify their reasoning process.

Contribution

The paper presents a novel dataset and user simulator for explainable multi-turn reasoning, along with a neural model that generates predictions and explanations.

Findings

Neural model achieves strong correlation between prediction accuracy and explanation quality.

e-QRAQ enables training and evaluation of explainable reasoning agents.

The dataset facilitates research in interpretable AI for complex reasoning tasks.

Abstract

In this paper we present a new dataset and user simulator e-QRAQ (explainable Query, Reason, and Answer Question) which tests an Agent's ability to read an ambiguous text; ask questions until it can answer a challenge question; and explain the reasoning behind its questions and answer. The User simulator provides the Agent with a short, ambiguous story and a challenge question about the story. The story is ambiguous because some of the entities have been replaced by variables. At each turn the Agent may ask for the value of a variable or try to answer the challenge question. In response the User simulator provides a natural language explanation of why the Agent's query or answer was useful in narrowing down the set of possible answers, or not. To demonstrate one potential application of the e-QRAQ dataset, we train a new neural architecture based on End-to-End Memory Networks to successfully generate both predictions and partial explanations of its current understanding of the problem. We observe a strong correlation between the quality of the prediction and explanation.