Entropy and relative entropy from information-theoretic principles

TL;DR

This paper develops an axiomatic framework for defining entropies and relative entropies based on minimal information-theoretic principles, establishing their fundamental properties and relationships.

Contribution

It introduces a minimal axiomatic approach that characterizes entropies and relative entropies and proves a one-to-one correspondence between them.

Findings

Relative entropies are bounded by Re9nyi divergences of order 0 and e9f8.

Conditions for positive definiteness of relative entropies are provided.

A one-to-one correspondence between entropies and relative entropies is established.

Abstract

We introduce an axiomatic approach to entropies and relative entropies that relies only on minimal information-theoretic axioms, namely monotonicity under mixing and data-processing as well as additivity for product distributions. We find that these axioms induce sufficient structure to establish continuity in the interior of the probability simplex and meaningful upper and lower bounds, e.g., we find that every relative entropy must lie between the R\'enyi divergences of order $0$ and $\infty$. We further show simple conditions for positive definiteness of such relative entropies and a characterisation in term of a variant of relative trumping. Our main result is a one-to-one correspondence between entropies and relative entropies.