Language modeling with Neural trans-dimensional random fields

TL;DR

This paper introduces neural trans-dimensional random field language models that leverage neural networks for nonlinear potentials, achieving improved performance and efficiency over previous models and LSTM LMs.

Contribution

It proposes neural TRFs with deep CNN potentials and a novel training strategy, combining neural network advantages with efficient inference in TRF frameworks.

Findings

Neural TRFs outperform discrete TRFs in speech recognition tasks.

Neural TRFs slightly outperform LSTM LMs with fewer parameters.

Neural TRFs offer 16x faster inference than LSTM models.

Abstract

Trans-dimensional random field language models (TRF LMs) have recently been introduced, where sentences are modeled as a collection of random fields. The TRF approach has been shown to have the advantages of being computationally more efficient in inference than LSTM LMs with close performance and being able to flexibly integrating rich features. In this paper we propose neural TRFs, beyond of the previous discrete TRFs that only use linear potentials with discrete features. The idea is to use nonlinear potentials with continuous features, implemented by neural networks (NNs), in the TRF framework. Neural TRFs combine the advantages of both NNs and TRFs. The benefits of word embedding, nonlinear feature learning and larger context modeling are inherited from the use of NNs. At the same time, the strength of efficient inference by avoiding expensive softmax is preserved. A number of technical contributions, including employing deep convolutional neural networks (CNNs) to define the potentials and incorporating the joint stochastic approximation (JSA) strategy in the training algorithm, are developed in this work, which enable us to successfully train neural TRF LMs. Various LMs are evaluated in terms of speech recognition WERs by rescoring the 1000-best lists of WSJ'92 test data. The results show that neural TRF LMs not only improve over discrete TRF LMs, but also perform slightly better than LSTM LMs with only one fifth of parameters and 16x faster inference efficiency.