"Noisier" Noise Contrastive Eestimation is (Almost) Maximum Likelihood

TL;DR

This paper introduces "Noisier" NCE, a modification that improves density-ratio estimation by artificially increasing noise magnitude, enabling faster convergence and better performance in high-dimensional, multimodal datasets.

Contribution

It presents a simple, computationally inexpensive tweak to NCE that aligns its gradient with MLE, enhancing its effectiveness in challenging density estimation tasks.

Findings

"Noisier" NCE achieves state-of-the-art results on CIFAR-10 and ImageNet64x64.

It produces high-quality 1-step and 10-step samplers.

Training is up to twice as fast compared to traditional methods.

Abstract

Noise Contrastive Estimation (NCE) has fueled major breakthroughs in representation learning and generative modeling. Yet a long-standing challenge remains: accurately estimating ratios between distributions that differ substantially, which significantly limits the applicability of NCE on modern high-dimensional and multimodal datasets. We revisit this problem from a less explored perspective: the magnitude of the noise distribution. Specifically, we show that with a virtually scaled (\ie, artificially increased) noise magnitude, the gradient of the NCE objective can closely align with that of Maximum Likelihood, enabling a trajectory-wise approximation from NCE to MLE, and faster convergence both theoretically and empirically. Building on this insight, we introduce ``Noisier'' NCE, a simple drop-in modification to vanilla NCE that incurs little to no extra computational cost, while effectively handling density-ratio estimation in challenging regimes where traditional MLE and NCE struggle. Beyond improving classical density-ratio learning, ``Noisier'' NCE proves broadly applicable: it achieves strong results across image modeling, anomaly detection, and offline black-box optimization. On CIFAR-10 and ImageNet64x64 datasets, it yields 10-step and even 1-step samplers that match or surpass state-of-the-art methods, while cutting training iterations by up to half.