Structure-aware divergences for comparing probability distributions

TL;DR

This paper introduces a new family of divergences that incorporate the structure of the domain to compare probability distributions more effectively, outperforming traditional methods in pattern detection and computational efficiency.

Contribution

The authors develop structure-aware divergences based on Bregman divergences of structure-aware entropies, enhancing the comparison of distributions with underlying domain relationships.

Findings

Recover planted patterns in synthetic clustering tasks

Significantly faster than optimal transport distances

Reproduce ecological and economic geographic insights

Abstract

Many natural and social science systems are described using probability distributions over elements that are related to each other: for instance, occupations with shared skills or species with similar traits. Standard information theory quantities such as entropies and $f$-divergences treat elements interchangeably and are blind to the similarity structure. We introduce a family of divergences that are sensitive to the geometry of the underlying domain. By virtue of being the Bregman divergences of structure-aware entropies, they provide a framework that retains several advantages of Kullback-Leibler divergence and Shannon entropy. Structure-aware divergences recover planted patterns in a synthetic clustering task that conventional divergences miss and are orders of magnitude faster than optimal transport distances. We demonstrate their applicability in economic geography and ecology, where structure plays an important role. Modelling different notions of occupation relatedness yields qualitatively different regionalisations of their geographic distribution. Our methods also reproduce established insights into functional $\beta$-diversity in ecology obtained with optimal transport methods.