Unification of field theory and maximum entropy methods for learning probability densities

TL;DR

This paper unifies maximum entropy and Bayesian field theory methods for estimating smooth probability densities from finite data, showing their equivalence in certain limits and providing a practical, rapid computational approach for one-dimensional data.

Contribution

It demonstrates that maximum entropy estimates are a limit case of Bayesian field theory, unifies the two approaches, and offers a practical method with software for 1D density estimation.

Findings

Maximum entropy estimates are recovered in the infinite smoothness limit of Bayesian field theory.

Bayesian field theory can estimate densities without boundary conditions.

The approach is computationally fast for one-dimensional data.

Abstract

The need to estimate smooth probability distributions (a.k.a. probability densities) from finite sampled data is ubiquitous in science. Many approaches to this problem have been described, but none is yet regarded as providing a definitive solution. Maximum entropy estimation and Bayesian field theory are two such approaches. Both have origins in statistical physics, but the relationship between them has remained unclear. Here I unify these two methods by showing that every maximum entropy density estimate can be recovered in the infinite smoothness limit of an appropriate Bayesian field theory. I also show that Bayesian field theory estimation can be performed without imposing any boundary conditions on candidate densities, and that the infinite smoothness limit of these theories recovers the most common types of maximum entropy estimates. Bayesian field theory is thus seen to provide a natural test of the validity of the maximum entropy null hypothesis. Bayesian field theory also returns a lower entropy density estimate when the maximum entropy hypothesis is falsified. The computations necessary for this approach can be performed rapidly for one-dimensional data, and software for doing this is provided. Based on these results, I argue that Bayesian field theory is poised to provide a definitive solution to the density estimation problem in one dimension.