Analyzing and Improving the Optimization Landscape of Noise-Contrastive Estimation

TL;DR

This paper investigates the optimization challenges of noise-contrastive estimation (NCE), formally analyzes the impact of noise distribution choices, and proposes a variant called eNCE with improved landscape properties.

Contribution

It provides a formal analysis of NCE's landscape issues and introduces eNCE, a variant with an exponential loss that improves optimization.

Findings

Poor noise distribution causes flat loss landscapes in NCE.

eNCE with exponential loss addresses landscape issues.

Normalized gradient descent effectively optimizes eNCE in certain cases.

Abstract

Noise-contrastive estimation (NCE) is a statistically consistent method for learning unnormalized probabilistic models. It has been empirically observed that the choice of the noise distribution is crucial for NCE's performance. However, such observations have never been made formal or quantitative. In fact, it is not even clear whether the difficulties arising from a poorly chosen noise distribution are statistical or algorithmic in nature. In this work, we formally pinpoint reasons for NCE's poor performance when an inappropriate noise distribution is used. Namely, we prove these challenges arise due to an ill-behaved (more precisely, flat) loss landscape. To address this, we introduce a variant of NCE called "eNCE" which uses an exponential loss and for which normalized gradient descent addresses the landscape issues provably when the target and noise distributions are in a given exponential family.