ReviewQA: a relational aspect-based opinion reading dataset

TL;DR

ReviewQA introduces a relational aspect-based dataset for hotel review question-answering, emphasizing reasoning over simple span extraction, with over 500,000 questions to benchmark model understanding and reasoning capabilities.

Contribution

The paper presents ReviewQA, a large-scale dataset designed to evaluate relational reasoning in question-answering models on hotel reviews.

Findings

Models show varied strengths across different reasoning tasks.

Benchmark results highlight gaps in current model reasoning abilities.

Dataset enables comprehensive evaluation of reasoning competencies.

Abstract

Deep reading models for question-answering have demonstrated promising performance over the last couple of years. However current systems tend to learn how to cleverly extract a span of the source document, based on its similarity with the question, instead of seeking for the appropriate answer. Indeed, a reading machine should be able to detect relevant passages in a document regarding a question, but more importantly, it should be able to reason over the important pieces of the document in order to produce an answer when it is required. To motivate this purpose, we present ReviewQA, a question-answering dataset based on hotel reviews. The questions of this dataset are linked to a set of relational understanding competencies that we expect a model to master. Indeed, each question comes with an associated type that characterizes the required competency. With this framework, it is possible to benchmark the main families of models and to get an overview of what are the strengths and the weaknesses of a given model on the set of tasks evaluated in this dataset. Our corpus contains more than 500.000 questions in natural language over 100.000 hotel reviews. Our setup is projective, the answer of a question does not need to be extracted from a document, like in most of the recent datasets, but selected among a set of candidates that contains all the possible answers to the questions of the dataset. Finally, we present several baselines over this dataset.