Noise contrastive estimation: asymptotics, comparison with MC-MLE

TL;DR

This paper establishes the asymptotic properties of noise contrastive estimation (NCE) for un-normalised models, compares it with Monte Carlo maximum likelihood estimation (MC-MLE), and demonstrates NCE's variance reduction advantages.

Contribution

It provides the first rigorous asymptotic analysis of NCE, compares it with MC-MLE under various regimes, and shows NCE's superior efficiency in certain settings.

Findings

NCE is asymptotically consistent and normal.

When artificial data points grow large, NCE and MC-MLE are asymptotically equivalent.

NCE has smaller asymptotic variance than MC-MLE when data points grow and their ratio converges.

Abstract

A statistical model is said to be un-normalised when its likelihood function involves an intractable normalising constant. Two popular methods for parameter inference for these models are MC-MLE (Monte Carlo maximum likelihood estimation), and NCE (noise contrastive estimation); both methods rely on simulating artificial data-points to approximate the normalising constant. While the asymptotics of MC-MLE have been established under general hypotheses (Geyer, 1994), this is not so for NCE. We establish consistency and asymptotic normality of NCE estimators under mild assumptions. We compare NCE and MC-MLE under several asymptotic regimes. In particular, we show that, when m goes to infinity while n is fixed (m and n being respectively the number of artificial data-points, and actual data-points), the two estimators are asymptotically equivalent. Conversely, we prove that, when the artificial data-points are IID, and when n goes to infinity while m/n converges to a positive constant, the asymptotic variance of a NCE estimator is always smaller than the asymptotic variance of the corresponding MC-MLE estimator. We illustrate the variance reduction brought by NCE through a numerical study.