Entropic Inference

TL;DR

This paper reviews entropic inference, emphasizing the role of information and Bayesian beliefs, and introduces the Maximum relative Entropy method as a unifying framework for inference.

Contribution

It demonstrates that the logarithmic relative entropy is the unique tool for inference, unifying MaxEnt and Bayesian methods into a single framework.

Findings

Maximum relative Entropy (ME) unifies MaxEnt and Bayesian inference.

Logarithmic relative entropy is identified as the unique inference tool.

The approach clarifies the epistemological foundations of information and belief updating.

Abstract

In this tutorial we review the essential arguments behing entropic inference. We focus on the epistemological notion of information and its relation to the Bayesian beliefs of rational agents. The problem of updating from a prior to a posterior probability distribution is tackled through an eliminative induction process that singles out the logarithmic relative entropy as the unique tool for inference. The resulting method of Maximum relative Entropy (ME), includes as special cases both MaxEnt and Bayes' rule, and therefore unifies the two themes of these workshops -- the Maximum Entropy and the Bayesian methods -- into a single general inference scheme.