Computing Divergences between Discrete Decomposable Models

TL;DR

This paper presents a method to compute exact divergences, including the alpha-beta family, between decomposable models like chordal Markov networks efficiently, leveraging their structure to overcome intractability in high dimensions.

Contribution

It introduces an approach to compute a broad class of divergences exactly between decomposable models, exploiting their structure for efficiency.

Findings

Exact divergence computation is feasible for decomposable models.

The method applies to divergences like KL, Hellinger, and chi-squared.

Computational complexity depends on the treewidth of the models.

Abstract

There are many applications that benefit from computing the exact divergence between 2 discrete probability measures, including machine learning. Unfortunately, in the absence of any assumptions on the structure or independencies within these distributions, computing the divergence between them is an intractable problem in high dimensions. We show that we are able to compute a wide family of functionals and divergences, such as the alpha-beta divergence, between two decomposable models, i.e. chordal Markov networks, in time exponential to the treewidth of these models. The alpha-beta divergence is a family of divergences that include popular divergences such as the Kullback-Leibler divergence, the Hellinger distance, and the chi-squared divergence. Thus, we can accurately compute the exact values of any of this broad class of divergences to the extent to which we can accurately model the two distributions using decomposable models.