Breaking the Softmax Bottleneck: A High-Rank RNN Language Model

TL;DR

This paper identifies a limitation in Softmax-based neural language models related to their expressiveness and proposes a simple method to overcome this bottleneck, significantly improving perplexity scores on multiple datasets.

Contribution

The paper introduces a novel approach to address the Softmax bottleneck, enhancing the capacity of language models to better capture natural language complexity.

Findings

Achieved state-of-the-art perplexities on Penn Treebank and WikiText-2 datasets.

Outperformed baseline models by over 5.6 perplexity points on the 1B Word dataset.

Demonstrated that the proposed method effectively increases model expressiveness.

Abstract

We formulate language modeling as a matrix factorization problem, and show that the expressiveness of Softmax-based models (including the majority of neural language models) is limited by a Softmax bottleneck. Given that natural language is highly context-dependent, this further implies that in practice Softmax with distributed word embeddings does not have enough capacity to model natural language. We propose a simple and effective method to address this issue, and improve the state-of-the-art perplexities on Penn Treebank and WikiText-2 to 47.69 and 40.68 respectively. The proposed method also excels on the large-scale 1B Word dataset, outperforming the baseline by over 5.6 points in perplexity.