Regularizing and Optimizing LSTM Language Models

TL;DR

This paper introduces new regularization and optimization techniques for LSTM language models, achieving state-of-the-art perplexities on standard datasets through weight dropout and a novel averaging method.

Contribution

It proposes the weight-dropped LSTM with DropConnect and NT-ASGD, improving regularization and optimization for language modeling.

Findings

Achieved state-of-the-art perplexity of 57.3 on Penn Treebank

Achieved state-of-the-art perplexity of 65.8 on WikiText-2

Further improved perplexity to 52.8 and 52.0 with neural cache

Abstract

Recurrent neural networks (RNNs), such as long short-term memory networks (LSTMs), serve as a fundamental building block for many sequence learning tasks, including machine translation, language modeling, and question answering. In this paper, we consider the specific problem of word-level language modeling and investigate strategies for regularizing and optimizing LSTM-based models. We propose the weight-dropped LSTM which uses DropConnect on hidden-to-hidden weights as a form of recurrent regularization. Further, we introduce NT-ASGD, a variant of the averaged stochastic gradient method, wherein the averaging trigger is determined using a non-monotonic condition as opposed to being tuned by the user. Using these and other regularization strategies, we achieve state-of-the-art word level perplexities on two data sets: 57.3 on Penn Treebank and 65.8 on WikiText-2. In exploring the effectiveness of a neural cache in conjunction with our proposed model, we achieve an even lower state-of-the-art perplexity of 52.8 on Penn Treebank and 52.0 on WikiText-2.