Strategies for Training Large Vocabulary Neural Language Models

TL;DR

This paper systematically compares various strategies for training large vocabulary neural language models, focusing on efficiency, accuracy, and handling rare words, and introduces improvements to existing methods.

Contribution

It provides a comprehensive evaluation of softmax, hierarchical softmax, sampling, NCE, and self normalization, including new extensions and an efficient softmax variant.

Findings

Self normalization can be effectively extended as a proper likelihood estimator.

The proposed efficient softmax reduces computational cost while maintaining accuracy.

Different strategies show varying trade-offs in speed and performance depending on the task.

Abstract

Training neural network language models over large vocabularies is still computationally very costly compared to count-based models such as Kneser-Ney. At the same time, neural language models are gaining popularity for many applications such as speech recognition and machine translation whose success depends on scalability. We present a systematic comparison of strategies to represent and train large vocabularies, including softmax, hierarchical softmax, target sampling, noise contrastive estimation and self normalization. We further extend self normalization to be a proper estimator of likelihood and introduce an efficient variant of softmax. We evaluate each method on three popular benchmarks, examining performance on rare words, the speed/accuracy trade-off and complementarity to Kneser-Ney.