Learning a Word-Level Language Model with Sentence-Level Noise Contrastive Estimation for Contextual Sentence Probability Estimation

TL;DR

This paper introduces a sentence-level noise-contrastive estimation method for word-level language models to better estimate the probability of sentences given prior context, improving contextual sentence probability estimation.

Contribution

The work extends sentence-level NCE training to contextual SPE, enabling better probability estimation conditioned on previous text, using a simple RNN model.

Findings

Improved sentence probability estimation accuracy.

Enhanced performance on multiple-choice cloze questions.

Demonstrated effectiveness of sentence-level NCE in contextual settings.

Abstract

Inferring the probability distribution of sentences or word sequences is a key process in natural language processing. While word-level language models (LMs) have been widely adopted for computing the joint probabilities of word sequences, they have difficulty in capturing a context long enough for sentence probability estimation (SPE). To overcome this, recent studies introduced training methods using sentence-level noise-contrastive estimation (NCE) with recurrent neural networks (RNNs). In this work, we attempt to extend it for contextual SPE, which aims to estimate a conditional sentence probability given a previous text. The proposed NCE samples negative sentences independently of a previous text so that the trained model gives higher probabilities to the sentences that are more consistent with \textcolor{blue}{the} context. We apply our method to a simple word-level RNN LM to focus on the effect of the sentence-level NCE training rather than on the network architecture. The quality of estimation was evaluated against multiple-choice cloze-style questions including both human and automatically generated questions. The experimental results show that the proposed method improved the SPE quality for the word-level RNN LM.