Learning probabilities from random observables in high dimensions: the maximum entropy distribution and others

TL;DR

This paper investigates how to learn a target probability distribution over binary variables from random observable expectations, analyzing the structure of the compatible distribution space and the effectiveness of maximum entropy methods.

Contribution

It introduces a biased measure over the version space, interpolating between unbiased and maximum entropy distributions, and uses the replica method to analyze phase transitions and learning efficiency.

Findings

Phase transitions improve target distribution learning at critical data amounts.

Maximum entropy distribution is not necessarily closer to the target than other compatible distributions.

Monte Carlo simulations confirm theoretical predictions for small systems.

Abstract

We consider the problem of learning a target probability distribution over a set of $N$ binary variables from the knowledge of the expectation values (with this target distribution) of $M$ observables, drawn uniformly at random. The space of all probability distributions compatible with these $M$ expectation values within some fixed accuracy, called version space, is studied. We introduce a biased measure over the version space, which gives a boost increasing exponentially with the entropy of the distributions and with an arbitrary inverse `temperature' $\Gamma$. The choice of $\Gamma$ allows us to interpolate smoothly between the unbiased measure over all distributions in the version space ($\Gamma=0$) and the pointwise measure concentrated at the maximum entropy distribution ($\Gamma \to \infty$). Using the replica method we compute the volume of the version space and other quantities of interest, such as the distance $R$ between the target distribution and the center-of-mass distribution over the version space, as functions of $\alpha=(\log M)/N$ and $\Gamma$ for large $N$. Phase transitions at critical values of $\alpha$ are found, corresponding to qualitative improvements in the learning of the target distribution and to the decrease of the distance $R$. However, for fixed $\alpha$, the distance $R$ does not vary with $\Gamma$, which means that the maximum entropy distribution is not closer to the target distribution than any other distribution compatible with the observable values. Our results are confirmed by Monte Carlo sampling of the version space for small system sizes ($N\le 10$).