KL Divergence Estimation with Multi-group Attribution

TL;DR

This paper introduces a new method for estimating KL divergence that accounts for sub-population contributions, ensuring fair and accurate divergence attribution across overlapping groups, with theoretical guarantees and improved empirical performance.

Contribution

It proposes the concept of multi-group attribution for KL divergence estimation, connecting it to multi-calibration, and demonstrates its effectiveness over existing methods.

Findings

Multi-group attribution improves divergence estimates conditioned on sub-populations.

The method ensures fairness by aligning divergence estimates with true subgroup differences.

Experimental results show superior performance compared to popular algorithms.

Abstract

Estimating the Kullback-Leibler (KL) divergence between two distributions given samples from them is well-studied in machine learning and information theory. Motivated by considerations of multi-group fairness, we seek KL divergence estimates that accurately reflect the contributions of sub-populations to the overall divergence. We model the sub-populations coming from a rich (possibly infinite) family $\mathcal{C}$ of overlapping subsets of the domain. We propose the notion of multi-group attribution for $\mathcal{C}$, which requires that the estimated divergence conditioned on every sub-population in $\mathcal{C}$ satisfies some natural accuracy and fairness desiderata, such as ensuring that sub-populations where the model predicts significant divergence do diverge significantly in the two distributions. Our main technical contribution is to show that multi-group attribution can be derived from the recently introduced notion of multi-calibration for importance weights [HKRR18, GRSW21]. We provide experimental evidence to support our theoretical results, and show that multi-group attribution provides better KL divergence estimates when conditioned on sub-populations than other popular algorithms.