Entropy, Information, and the Updating of Probabilities

TL;DR

This paper reviews the maximum entropy method as a universal inference framework that unifies Bayesian and entropic approaches, emphasizing its pragmatic derivation and broad applicability.

Contribution

It introduces the ME method as a unified approach for updating probabilities, encompassing MaxEnt and Bayes' rule, and highlights its ability to handle arbitrary priors and constraints.

Findings

The logarithmic relative entropy is identified as the unique universal updating tool.

The ME method unifies entropic and Bayesian inference.

It extends beyond single posteriors to quantify the probability of other distributions.

Abstract

This paper is a review of a particular approach to the method of maximum entropy as a general framework for inference. The discussion emphasizes the pragmatic elements in the derivation. An epistemic notion of information is defined in terms of its relation to the Bayesian beliefs of ideally rational agents. The method of updating from a prior to a posterior probability distribution is designed through an eliminative induction process. The logarithmic relative entropy is singled out as the unique tool for updating that (a) is of universal applicability; (b) that recognizes the value of prior information; and (c) that recognizes the privileged role played by the notion of independence in science. The resulting framework -- the ME method -- can handle arbitrary priors and arbitrary constraints. It includes MaxEnt and Bayes' rule as special cases and, therefore, it unifies entropic and Bayesian methods into a single general inference scheme. The ME method goes beyond the mere selection of a single posterior, but also addresses the question of how much less probable other distributions might be, which provides a direct bridge to the theories of fluctuations and large deviations.