Inferring a property of a large system from a small number of samples

TL;DR

This paper introduces a general framework for selecting priors in Bayesian inference of properties in large stochastic systems, using a linear combination of maximum entropy priors, which adapts based on data and improves estimation accuracy.

Contribution

The authors propose a unified method to select priors for arbitrary properties, reducing reliance on handcrafted priors and connecting Bayesian inference with statistical mechanics.

Findings

The method performs well compared to existing ad-hoc priors.

Often only one component of the prior expansion is needed for accurate inference.

The approach is validated on several example properties.

Abstract

Inferring the value of a property of a large stochastic system is a difficult task when the number of samples is insufficient to reliably estimate the probability distribution. The Bayesian estimator of the property of interest requires the knowledge of the prior distribution, and in many situations, it is not clear which prior should be used. Several estimators have been developed so far, in which the proposed prior was individually tailored for each property of interest; such is the case, for example, for the entropy, the amount of mutual information, or the correlation between pairs of variables. In this paper we propose a general framework to select priors, valid for arbitrary properties. We first demonstrate that only certain aspects of the prior distribution actually affect the inference process. We then expand the sought prior as a linear combination of a one-dimensional family of indexed priors, each of which is obtained through a maximum entropy approach with constrained mean value of the property under study. In many cases of interest, only one or very few components of the expansion turn out to contribute to the Bayesian estimator, so it is often valid to only keep a single component. The relevant component is selected by the data, so no handcrafted priors are required. We test the performance of this approximation with a few paradigmatic examples, and show that it performs well in comparison to the ad-hoc methods previously proposed in the literature. Our method highlights the connection between Bayesian inference and equilibrium statistical mechanics, since the most relevant component of the expansion can be argued to be the one with the right temperature.