Understanding Phase Transitions via Mutual Information and MMSE

TL;DR

This paper explores phase transitions in high-dimensional linear models using mutual information and MMSE, providing a rigorous proof for the replica method's predictions of optimal inference performance.

Contribution

It offers a tutorial on the linear model's information-theoretic analysis and proves the accuracy of the replica prediction for phase transitions.

Findings

Replica method predictions are proven exact for the linear model.

Phase transitions significantly affect inference quality.

The paper clarifies the connection between information theory and high-dimensional inference.

Abstract

The ability to understand and solve high-dimensional inference problems is essential for modern data science. This article examines high-dimensional inference problems through the lens of information theory and focuses on the standard linear model as a canonical example that is both rich enough to be practically useful and simple enough to be studied rigorously. In particular, this model can exhibit phase transitions where an arbitrarily small change in the model parameters can induce large changes in the quality of estimates. For this model, the performance of optimal inference can be studied using the replica method from statistical physics but, until recently, it was not known if the resulting formulas were actually correct. In this chapter, we present a tutorial description of the standard linear model and its connection to information theory. We also describe the replica prediction for this model and outline the authors' recent proof that it is exact.