# On the correspondence between thermodynamics and inference

## TL;DR

This paper draws an analogy between thermodynamics and Bayesian inference, introducing new concepts like learning capacity and Gibbs entropy to better understand learning mechanisms and define objective priors.

## Contribution

It introduces the concepts of learning capacity and Gibbs entropy, providing new insights into learning behavior and a novel approach to defining uninformative priors in Bayesian inference.

## Key findings

- Learning capacity explains anomalously-high model performance.
- Gibbs entropy offers a way to count distinguishable distributions.
- The generalized principle of indifference defines objective priors.

## Abstract

We expand upon a natural analogy between Bayesian statistics and statistical physics in which sample size corresponds to inverse temperature. This analogy motivates the definition of two novel statistical quantities: a learning capacity and a Gibbs entropy. The analysis of the learning capacity, corresponding to the heat capacity in thermal physics, leads to new insight into the mechanism of learning and explains why some models have anomalously-high learning performance. We explore the properties of the learning capacity in a number of examples, including a sloppy model. Next, we propose that the Gibbs entropy provides a natural device for counting distinguishable distributions in the context of Bayesian inference. We use this device to define a generalized principle of indifference (GPI) in which every distinguishable model is assigned equal a priori probability. This principle results in a new solution to a long-standing problem in Bayesian inference: the definition of an objective or uninformative prior. A key characteristic of this new approach is that it can be applied to analyses where the model dimension is unknown and circumvents the automatic rejection of higher-dimensional models in Bayesian inference.