Long-Short Range Context Neural Networks for Language Modeling

TL;DR

This paper introduces a novel neural network architecture that separately models short and long-range context in language modeling, leading to significant perplexity improvements over existing methods.

Contribution

The paper proposes the Long-Short Range Context (LSRC) network, an adaptation of LSTM that explicitly captures and merges short and long-range dependencies for improved language modeling.

Findings

Significant perplexity reduction on PTB and LTCB datasets.

Effective modeling of both local and global linguistic contexts.

Outperforms state-of-the-art language models.

Abstract

The goal of language modeling techniques is to capture the statistical and structural properties of natural languages from training corpora. This task typically involves the learning of short range dependencies, which generally model the syntactic properties of a language and/or long range dependencies, which are semantic in nature. We propose in this paper a new multi-span architecture, which separately models the short and long context information while it dynamically merges them to perform the language modeling task. This is done through a novel recurrent Long-Short Range Context (LSRC) network, which explicitly models the local (short) and global (long) context using two separate hidden states that evolve in time. This new architecture is an adaptation of the Long-Short Term Memory network (LSTM) to take into account the linguistic properties. Extensive experiments conducted on the Penn Treebank (PTB) and the Large Text Compression Benchmark (LTCB) corpus showed a significant reduction of the perplexity when compared to state-of-the-art language modeling techniques.