Mutual Information, Information-Theoretic Thresholds and the Condensation Phenomenon at Positive Temperature

TL;DR

This paper investigates the asymptotic behavior of mutual information, relative entropy, and free entropy in complex probabilistic models using factor graphs, revealing structural similarities across various disciplines.

Contribution

It establishes the limits of mutual information, relative entropy, and free entropy in sparse graph models, unifying several concepts across different fields.

Findings

Limit of mutual information established

Limit of relative entropy determined

Conjectured limit of quenched free entropy proved

Abstract

There is a vast body of recent literature on the reliability of communication through noisy channels, the recovery of community structures in the stochastic block model, the limiting behavior of the free entropy in spin glasses and the solution space structure of constraint satisfaction problems. At first glance, these topics ranging across several disciplines might seem unrelated. However, taking a closer look, structural similarities can be easily identified. Factor graphs exploit these similarities to model the aforementioned objects and concepts in a unified manner. In this contribution we discuss the asymptotic average case behavior of several quantities, where the average is taken over sparse Erd\H{o}s-R\'enyi type (hyper-) graphs with positive weights, under certain assumptions. For one, we establish the limit of the mutual information, which is used in coding theory to measure the reliability of communication. We also determine the limit of the relative entropy, which can be used to decide if weak recovery is possible in the stochastic block model. Further, we prove the conjectured limit of the quenched free entropy over the planted ensemble, which we use to obtain the preceding limits. Finally, we describe the asymptotic behavior of the quenched free entropy (over the null model) in terms of the limiting relative entropy.