Condensation Transition in Entropy-Constrained Probability Spaces

TL;DR

This paper investigates the structure of high-dimensional probability spaces constrained by fixed Shannon entropy, revealing a phase transition where distributions condense into dominant states below a critical entropy.

Contribution

It introduces a discretization method and demonstrates a condensation phase transition in probability spaces constrained by entropy, linking to phenomena in machine learning and ecology.

Findings

A critical entropy scale H_c ~ log K - 1 + γ is identified.

Below H_c, most distributions are condensed with one dominant component.

The results connect entropy constraints to natural sparsity and dominance phenomena.

Abstract

The organization of high-dimensional probability spaces is a fundamental problem at the intersection of statistical physics and information theory. Here, we analyze the distributions populating level surfaces of the probability simplex $\Delta_{K-1}$ defined by a fixed Shannon entropy. We introduce a discretization strategy that assigns equal statistical weight to distinct microstate distributions and enables a combinatorial analysis of the simplex. A condensation phase transition is shown to take place below a critical entropy that scales as $H_c \simeq \log K - 1 + \gamma$ in the thermodynamic limit. For entropy values $H_0 < H_c$, the overwhelming majority of distributions are found in a condensed state, in which a single component captures a macroscopic fraction of the total probability mass while the remaining components form a homogeneous fluid background. These results provide a framework for understanding phenomena such as overconfident predictions in machine learning and the emergence of dominant species in ecology, and suggest that sparsity can arise naturally from entropic constraints in high-dimensional manifolds.