Discrete Bridges for Mutual Information Estimation

TL;DR

This paper introduces a novel Discrete Bridge Mutual Information (DBMI) estimator that leverages discrete diffusion bridge models to accurately estimate mutual information between discrete variables, especially in challenging data settings.

Contribution

The paper presents a new MI estimator based on discrete diffusion bridges, addressing limitations of traditional methods for discrete data and framing MI estimation as a domain transfer problem.

Findings

Effective in low-dimensional MI estimation

Performs well on image-based data

Addresses challenges of conventional MI estimators

Abstract

Diffusion bridge models in both continuous and discrete state spaces have recently become powerful tools in the field of generative modeling. In this work, we leverage the discrete state space formulation of bridge matching models to address another important problem in machine learning and information theory: the estimation of the mutual information (MI) between discrete random variables. By neatly framing MI estimation as a domain transfer problem, we construct a Discrete Bridge Mutual Information (DBMI) estimator suitable for discrete data, which poses difficulties for conventional MI estimators. We showcase the performance of our estimator on two MI estimation settings: low-dimensional and image-based.