A Simple Language Model based on PMI Matrix Approximations

TL;DR

This paper proposes a simple yet effective language modeling approach that estimates PMI matrices using modified word2vec training, resulting in models closely related to NCE-based methods.

Contribution

It introduces a PMI-based language model derived from a slight modification of the word2vec algorithm, connecting word embeddings with probabilistic language modeling.

Findings

Models trained with negative sampling effectively estimate PMI matrices.

The approach yields principled language models comparable to NCE-based methods.

Simple modifications to word2vec can produce robust language models.

Abstract

In this study, we introduce a new approach for learning language models by training them to estimate word-context pointwise mutual information (PMI), and then deriving the desired conditional probabilities from PMI at test time. Specifically, we show that with minor modifications to word2vec's algorithm, we get principled language models that are closely related to the well-established Noise Contrastive Estimation (NCE) based language models. A compelling aspect of our approach is that our models are trained with the same simple negative sampling objective function that is commonly used in word2vec to learn word embeddings.