Partition function approach to non-Gaussian likelihoods: information theory and state variables for Bayesian inference

TL;DR

This paper bridges statistical physics and Bayesian inference by interpreting sampling and updates through thermodynamic concepts, introducing a partition function approach to non-Gaussian likelihoods and proposing measures of system complexity.

Contribution

It introduces a thermodynamics-inspired framework for Bayesian inference, applying partition functions and information theory to analyze sampling processes and system complexity.

Findings

Thermodynamic analogies clarify Bayesian sampling and updates.

Partition functions provide a new analytical tool for non-Gaussian likelihoods.

Effective dimension measures system complexity in inference processes.

Abstract

The significance of statistical physics concepts such as entropy extends far beyond classical thermodynamics. We interpret the similarity between partitions in statistical mechanics and partitions in Bayesian inference as an articulation of a result by Jaynes (1957), who clarified that thermodynamics is in essence a theory of information. In this, every sampling process has a mechanical analogue. Consequently, the divide between ensembles of samplers in parameter space and sampling from a mechanical system in thermodynamic equilibrium would be artificial. Based on this realisation, we construct a continuous modelling of a Bayes update akin to a transition between thermodynamic ensembles. This leads to an information theoretic interpretation of Jazinsky's equality, relating the expenditure of work to the influence of data via the likelihood. We propose one way to transfer the vocabulary and the formalism of thermodynamics (energy, work, heat) and statistical mechanics (partition functions) to statistical inference, starting from Bayes' law. Different kinds of inference processes are discussed and relative entropies are shown to follow from suitably constructed partitions as an analytical formulation of sampling processes. Lastly, we propose an effective dimension as a measure of system complexity. A numerical example from cosmology is put forward to illustrate these results.