Linguistic Knowledge as Memory for Recurrent Neural Networks

TL;DR

This paper introduces a method to incorporate external linguistic knowledge as explicit memory in recurrent neural networks, improving their ability to model long-term dependencies and achieve state-of-the-art results in text comprehension tasks.

Contribution

The paper presents a novel approach that encodes external linguistic knowledge as graph-based memory in RNNs, enhancing their performance on coreference and comprehension tasks.

Findings

Achieved state-of-the-art results on CNN, bAbi, and LAMBADA benchmarks.

Solved 15 out of 20 bAbi QA tasks with limited training data.

Demonstrated the model's ability to encode detailed entity information.

Abstract

Training recurrent neural networks to model long term dependencies is difficult. Hence, we propose to use external linguistic knowledge as an explicit signal to inform the model which memories it should utilize. Specifically, external knowledge is used to augment a sequence with typed edges between arbitrarily distant elements, and the resulting graph is decomposed into directed acyclic subgraphs. We introduce a model that encodes such graphs as explicit memory in recurrent neural networks, and use it to model coreference relations in text. We apply our model to several text comprehension tasks and achieve new state-of-the-art results on all considered benchmarks, including CNN, bAbi, and LAMBADA. On the bAbi QA tasks, our model solves 15 out of the 20 tasks with only 1000 training examples per task. Analysis of the learned representations further demonstrates the ability of our model to encode fine-grained entity information across a document.