InfoBridge: Mutual Information estimation via Bridge Matching
Sergei Kholkin, Ivan Butakov, Evgeny Burnaev, Nikita Gushchin, Alexander Korotin
TL;DR
This paper introduces InfoBridge, a novel mutual information estimator using diffusion bridge models, which effectively handles challenging data scenarios and outperforms traditional methods on various benchmarks and real-world datasets.
Contribution
The paper presents a new MI estimation method based on diffusion bridge models, framing MI estimation as a domain transfer problem for unbiased and robust estimation.
Findings
Performs well on low-dimensional, image-based, and high MI benchmarks.
Achieves accurate MI estimation on protein language model embeddings.
Outperforms conventional MI estimators in challenging data scenarios.
Abstract
Diffusion bridge models have recently become a powerful tool in the field of generative modeling. In this work, we leverage their power to address another important problem in machine learning and information theory, the estimation of the mutual information (MI) between two random variables. Neatly framing MI estimation as a domain transfer problem, we construct an unbiased estimator for data posing difficulties for conventional MI estimators. We showcase the performance of our estimator on three standard MI estimation benchmarks, i.e., low-dimensional, image-based and high MI, and on real-world data, i.e., protein language model embeddings.
Peer Reviews
Decision·ICLR 2026 Poster
+ The main contribution is much sought after: An unbiased, low variance estimate of mutual information has impact across fields. + The paper's literature review is apt and does a great job motivating the problem. + The paper is full of illuminating insights tying information theory, diffusion models and stochastic calculus.
+ The evaluation is solely conducted by measuring the discrepancy from a ground truth mutual information. This is sound an necessary but also limited by itself. The paper would be more convincing if the proposed estimate of the mutual information were used as an objective, or its learned representations evaluated on downstream tasks. + The work could be made much more accessible. The background section runs two full pages, and the authors only introduce their method in page 5. We suggest the m
* Conceptual novelty: Clear, principled decomposition of MI into a time-integral of drift differences between two bridge processes; different viewpoint than score-based MINDE. * Methodological soundness: Builds on established conditional bridge matching; training and estimation procedures are straightforward (Alg. 1–2). * Practicality: Uses only joint samples and Brownian-bridge simulation (no explicit density estimation), which is attractive for complex data like images/embeddings.
* Unclear dependence on ϵ: The estimator formula scales with 1/(2ϵ). Please analyze or empirically demonstrate ϵ-invariance of the resulting MI and give guidance for choosing ϵ. * Bias/variance in practice: “Unbiased” is only in the idealized limit. Did you check the empirical calibration with confidence intervals vs. ground-truth MI? Did you check sensitivity to network size, training steps, and t-discretization? * Independence construction: The independent process uses permuted pairs (x_0,\h
- The proposed estimator relies on diffusion bridge models, thus leveraging a new technique for MI estimation - The proposed generative MI estimator is unbiased, differently from MINDE which is biased - The experimental results show the effectiveness of the proposed estimator
- Although the proposed estimator leverages diffusion bridge models, which differ from diffusion models, the proposed framework appears as an extension of MINDE - The paper is not well-written. The clarity could definitively be improved. For instance, SDE is not defined (stochastic differential equation). The equation in line 145 should be written in two lines, as it exceeds the borders of the paper. There are many typos: line 96 (conditioned its on values), line 264 (as a domain transfer task),
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsAdvanced Database Systems and Queries · Data Management and Algorithms · Data Quality and Management
MethodsDiffusion