Compositional Language Understanding with Text-based Relational Reasoning

TL;DR

This paper introduces a new benchmark dataset for evaluating neural networks' ability to perform relational reasoning in language understanding, highlighting the importance of relational inductive biases for better generalization.

Contribution

The paper presents a novel benchmark dataset for relational reasoning in language understanding and demonstrates that message-passing neural networks outperform traditional RNNs on this task.

Findings

Message-passing models show superior combinatorial generalization.

Relational inductive bias improves reasoning performance.

Benchmark isolates relational reasoning capabilities.

Abstract

Neural networks for natural language reasoning have largely focused on extractive, fact-based question-answering (QA) and common-sense inference. However, it is also crucial to understand the extent to which neural networks can perform relational reasoning and combinatorial generalization from natural language---abilities that are often obscured by annotation artifacts and the dominance of language modeling in standard QA benchmarks. In this work, we present a novel benchmark dataset for language understanding that isolates performance on relational reasoning. We also present a neural message-passing baseline and show that this model, which incorporates a relational inductive bias, is superior at combinatorial generalization compared to a traditional recurrent neural network approach.