Predictive Representation Learning for Language Modeling

TL;DR

This paper introduces Predictive Representation Learning (PRL), a method that explicitly trains LSTMs to encode specific predictions, improving language modeling performance, convergence speed, and data efficiency.

Contribution

It proposes a novel PRL approach that constrains LSTMs to encode explicit predictions, enhancing language model effectiveness compared to traditional methods.

Findings

PRL significantly improves language modeling results

PRL leads to faster convergence of models

PRL performs better with limited data

Abstract

To effectively perform the task of next-word prediction, long short-term memory networks (LSTMs) must keep track of many types of information. Some information is directly related to the next word's identity, but some is more secondary (e.g. discourse-level features or features of downstream words). Correlates of secondary information appear in LSTM representations even though they are not part of an \emph{explicitly} supervised prediction task. In contrast, in reinforcement learning (RL), techniques that explicitly supervise representations to predict secondary information have been shown to be beneficial. Inspired by that success, we propose Predictive Representation Learning (PRL), which explicitly constrains LSTMs to encode specific predictions, like those that might need to be learned implicitly. We show that PRL 1) significantly improves two strong language modeling methods, 2) converges more quickly, and 3) performs better when data is limited. Our work shows that explicitly encoding a simple predictive task facilitates the search for a more effective language model.