Bregman divergence as general framework to estimate unnormalized statistical models

TL;DR

This paper introduces Bregman divergence as a comprehensive framework for estimating unnormalized statistical models, unifying various existing methods and exploring their interconnections and extensions.

Contribution

It demonstrates that noise-contrastive, ratio matching, and score matching are special cases within the Bregman divergence framework, providing new insights into their relationships.

Findings

Unified view of estimation methods via Bregman divergence

Connections between supervised and unsupervised learning techniques

Insights into boosting for unsupervised model estimation

Abstract

We show that the Bregman divergence provides a rich framework to estimate unnormalized statistical models for continuous or discrete random variables, that is, models which do not integrate or sum to one, respectively. We prove that recent estimation methods such as noise-contrastive estimation, ratio matching, and score matching belong to the proposed framework, and explain their interconnection based on supervised learning. Further, we discuss the role of boosting in unsupervised learning.