R\'enyi Divergence and Majorization

TL;DR

This paper reviews the properties of Re9nyi divergence, explores its relation to majorization in continuous settings, and discusses its application in quantifying the effort to guess random sequences.

Contribution

It provides a comprehensive review of Re9nyi divergence, extends majorization theory to continuous cases, and links divergence measures to information-gathering processes.

Findings

Re9nyi divergence shares key properties with information divergence.

Majorization theory is generalized from finite to continuous domains using Re9nyi divergence.

Re9nyi divergence quantifies the number of binary questions needed to identify a sequence.

Abstract

R\'enyi divergence is related to R\'enyi entropy much like information divergence (also called Kullback-Leibler divergence or relative entropy) is related to Shannon's entropy, and comes up in many settings. It was introduced by R\'enyi as a measure of information that satisfies almost the same axioms as information divergence. We review the most important properties of R\'enyi divergence, including its relation to some other distances. We show how R\'enyi divergence appears when the theory of majorization is generalized from the finite to the continuous setting. Finally, R\'enyi divergence plays a role in analyzing the number of binary questions required to guess the values of a sequence of random variables.