PMI Matrix Approximations with Applications to Neural Language Modeling

TL;DR

This paper demonstrates that negative sampling, previously thought unsuitable for language modeling, can be effectively used for this purpose through a novel matrix approximation approach, achieving competitive results.

Contribution

It provides a principled derivation and analysis of NEG-based language modeling, unifying word embedding and language modeling under a simplified objective.

Findings

NEG-based language models achieve comparable perplexity to NCE models.

The approach offers a unified framework for word embedding and language modeling.

Experimental results show a small advantage of NEG over NCE.

Abstract

The negative sampling (NEG) objective function, used in word2vec, is a simplification of the Noise Contrastive Estimation (NCE) method. NEG was found to be highly effective in learning continuous word representations. However, unlike NCE, it was considered inapplicable for the purpose of learning the parameters of a language model. In this study, we refute this assertion by providing a principled derivation for NEG-based language modeling, founded on a novel analysis of a low-dimensional approximation of the matrix of pointwise mutual information between the contexts and the predicted words. The obtained language modeling is closely related to NCE language models but is based on a simplified objective function. We thus provide a unified formulation for two main language processing tasks, namely word embedding and language modeling, based on the NEG objective function. Experimental results on two popular language modeling benchmarks show comparable perplexity results, with a small advantage to NEG over NCE.