A Unified View on Learning Unnormalized Distributions via Noise-Contrastive Estimation

TL;DR

This paper offers a unified perspective on noise-contrastive estimation (NCE) for learning unnormalized distributions, providing new theoretical insights and convergence rate results for exponential families.

Contribution

It unifies various existing methods for unnormalized distribution learning under the NCE framework and establishes their finite-sample convergence rates.

Findings

Unified view of NCE and related methods

Finite-sample convergence rates for exponential families

New regularity assumptions for estimator analysis

Abstract

This paper studies a family of estimators based on noise-contrastive estimation (NCE) for learning unnormalized distributions. The main contribution of this work is to provide a unified perspective on various methods for learning unnormalized distributions, which have been independently proposed and studied in separate research communities, through the lens of NCE. This unified view offers new insights into existing estimators. Specifically, for exponential families, we establish the finite-sample convergence rates of the proposed estimators under a set of regularity assumptions, most of which are new.