Unification of Maximum Entropy and Bayesian Inference via Plausible Reasoning

TL;DR

This paper introduces the Optimum Information Principle, unifying maximum entropy and Bayesian inference from first principles, explaining why entropy maximization is justified under incomplete information.

Contribution

It derives key inference rules like Bayes's rule and maximum likelihood from modified axioms of plausible reasoning, unifying them under a new foundational framework.

Findings

Derivation of minimum relative entropy principle from first principles

Unified framework for Bayesian inference and maximum entropy

Provides philosophical justification for entropy maximization

Abstract

This paper modifies Jaynes's axioms of plausible reasoning and derives the minimum relative entropy principle, Bayes's rule, as well as maximum likelihood from first principles. The new axioms, which I call the Optimum Information Principle, is applicable whenever the decision maker is given the data and the relevant background information. These axioms provide an answer to the question "why maximize entropy when faced with incomplete information?"