Improved training of neural trans-dimensional random field language models with dynamic noise-contrastive estimation

TL;DR

This paper introduces dynamic noise-contrastive estimation (DNCE) to improve training efficiency and reduce overfitting in neural trans-dimensional random field language models, enabling large-scale training and competitive performance.

Contribution

The paper proposes DNCE, an extension of NCE, which trains a dynamic noise distribution and interpolates data and noise, improving training efficiency and overfitting mitigation for neural TRF language models.

Findings

Neural TRF LMs perform as well as LSTM LMs with fewer parameters.

DNCE reduces training cost by decreasing the required noise samples.

Neural TRF LMs are 5x to 114x faster in rescoring tasks.

Abstract

A new whole-sentence language model - neural trans-dimensional random field language model (neural TRF LM), where sentences are modeled as a collection of random fields, and the potential function is defined by a neural network, has been introduced and successfully trained by noise-contrastive estimation (NCE). In this paper, we extend NCE and propose dynamic noise-contrastive estimation (DNCE) to solve the two problems observed in NCE training. First, a dynamic noise distribution is introduced and trained simultaneously to converge to the data distribution. This helps to significantly cut down the noise sample number used in NCE and reduce the training cost. Second, DNCE discriminates between sentences generated from the noise distribution and sentences generated from the interpolation of the data distribution and the noise distribution. This alleviates the overfitting problem caused by the sparseness of the training set. With DNCE, we can successfully and efficiently train neural TRF LMs on large corpus (about 0.8 billion words) with large vocabulary (about 568 K words). Neural TRF LMs perform as good as LSTM LMs with less parameters and being 5x~114x faster in rescoring sentences. Interpolating neural TRF LMs with LSTM LMs and n-gram LMs can further reduce the error rates.