The relation of bias with risk in empirically constrained inferences

TL;DR

This paper explores the connection between maximum entropy methods and Bayes optimal classifiers, demonstrating that maximum entropy can serve as a universally optimal decision rule under certain data constraints, with extensions to uncertain observations.

Contribution

It establishes a theoretical link between maximum entropy principles and Bayesian decision theory, extending results to cases with uncertain data constraints.

Findings

Maximum entropy is shown to be a universally Bayes optimal decision rule under data constraints.

The paper generalizes Sanov's theorem to handle uncertainty in observed expected losses.

Theoretical results connect asymptotic characterizations of entropy measures to classifier optimality.

Abstract

We give some results relating asymptotic characterisations of maximum entropy probability measures to characterisations of Bayes optimal classifiers. Our main theorems show that maximum entropy is a universally Bayes optimal decision rule given constraints on one's knowledge about some observed data in terms of an expected loss. We will extend this result to the case of uncertainty in the observations of expected losses by generalising Sanov's theorem to distributions of constraint values.