Contrastive Bayesian Inference for Unnormalized Models

TL;DR

This paper introduces a fully Bayesian method for unnormalized models that leverages noise contrastive estimation and Pólya-Gamma augmentation, enabling accurate inference and uncertainty quantification without tuning.

Contribution

It proposes a novel Bayesian framework that treats the normalizing constant as an unknown parameter, avoiding the need for likelihood tuning in unnormalized models.

Findings

Accurate point estimation demonstrated on simulated data.

Effective uncertainty quantification shown in real-data applications.

Gibbs sampler simplifies inference for exponential family models.

Abstract

Unnormalized (or energy-based) models provide a flexible framework for capturing the characteristics of data with complex dependency structures. However, the application of standard Bayesian inference methods has been severely limited because the parameter-dependent normalizing constant is either analytically intractable or computationally prohibitive to evaluate. A promising approach is score-based generalized Bayesian inference, which avoids evaluating the normalizing constant by replacing the likelihood with a scoring rule. However, this approach requires careful tuning of the likelihood information, and it may fail to yield valid inference without appropriate control. To overcome this difficulty, we propose a fully Bayesian framework for inference on unnormalized models that does not require such tuning. We build on noise contrastive estimation, which recasts inference as a binary classification problem between observed and noise samples, and treat the normalizing constant as an additional unknown parameter within the resulting likelihood. For exponential families, the classification likelihood becomes conditionally Gaussian via P\'olya-Gamma data augmentation, leading to a simple Gibbs sampler. We demonstrate the proposed approach through two models: time-varying density models for temporal point process data and sparse torus graph models for multivariate circular data. Through simulation studies and real-data analyses, the proposed method provides accurate point estimation and enables principled uncertainty quantification.