On the long-term learning ability of LSTM LMs

TL;DR

This paper investigates the long-term learning capabilities of LSTM language models by evaluating a contextual extension and analyzing its impact on sentence- and discourse-level models across text and speech.

Contribution

It introduces a contextual extension based on CBOW for LSTM LMs and analyzes its effectiveness at different levels, revealing insights into their reliance on contextual information.

Findings

Sentence-level models with the extension perform comparably to vanilla models.

The extension does not improve discourse-level models.

Discourse-level LSTM LMs already utilize contextual information effectively.

Abstract

We inspect the long-term learning ability of Long Short-Term Memory language models (LSTM LMs) by evaluating a contextual extension based on the Continuous Bag-of-Words (CBOW) model for both sentence- and discourse-level LSTM LMs and by analyzing its performance. We evaluate on text and speech. Sentence-level models using the long-term contextual module perform comparably to vanilla discourse-level LSTM LMs. On the other hand, the extension does not provide gains for discourse-level models. These findings indicate that discourse-level LSTM LMs already rely on contextual information to perform long-term learning.