co-BPM: a Bayesian Model for Divergence Estimation

TL;DR

co-BPM introduces a Bayesian approach that jointly models two distributions to accurately estimate their divergence, improving over traditional methods and applicable to various machine learning tasks.

Contribution

The paper presents co-BPM, a novel Bayesian model that learns coupled binary partitions for divergence estimation, avoiding independent density estimation pitfalls.

Findings

Accurately estimates multiple divergence types.

Outperforms state-of-the-art methods in simulations.

Effective on real-world data examples.

Abstract

Divergence is not only an important mathematical concept in information theory, but also applied to machine learning problems such as low-dimensional embedding, manifold learning, clustering, classification, and anomaly detection. We proposed a bayesian model---co-BPM---to characterize the discrepancy of two sample sets, i.e., to estimate the divergence of their underlying distributions. In order to avoid the pitfalls of plug-in methods that estimate each density independently, our bayesian model attempts to learn a coupled binary partition of the sample space that best captures the landscapes of both distributions, then make direct inference on their divergences. The prior is constructed by leveraging the sequential buildup of the coupled binary partitions and the posterior is sampled via our specialized MCMC. Our model provides a unified way to estimate various types of divergences and enjoys convincing accuracy. We demonstrate its effectiveness through simulations, comparisons with the \emph{state-of-the-art} and a real data example.