Revisiting Simple Neural Probabilistic Language Models

TL;DR

This paper revisits the neural probabilistic language model from 2003, demonstrating its surprising effectiveness on modern benchmarks and proposing a hybrid model that improves perplexity by combining it with Transformer architectures.

Contribution

The paper shows that the simple NPLM performs well on current benchmarks and introduces a hybrid Transformer-NPLM model that enhances language modeling performance.

Findings

NPLM outperforms expectations on modern benchmarks.

Replacing Transformer's first layer with NPLM's layer reduces perplexity.

Hybrid model shows consistent improvements across datasets.

Abstract

Recent progress in language modeling has been driven not only by advances in neural architectures, but also through hardware and optimization improvements. In this paper, we revisit the neural probabilistic language model (NPLM) of~\citet{Bengio2003ANP}, which simply concatenates word embeddings within a fixed window and passes the result through a feed-forward network to predict the next word. When scaled up to modern hardware, this model (despite its many limitations) performs much better than expected on word-level language model benchmarks. Our analysis reveals that the NPLM achieves lower perplexity than a baseline Transformer with short input contexts but struggles to handle long-term dependencies. Inspired by this result, we modify the Transformer by replacing its first self-attention layer with the NPLM's local concatenation layer, which results in small but consistent perplexity decreases across three word-level language modeling datasets.