Two Measures of Dependence

TL;DR

This paper introduces two new dependence measures between random variables based on Rénnyi divergence and relative α-entropy, generalizing mutual information and exploring their properties and applications.

Contribution

It presents two novel dependence measures that extend mutual information using Rénnyi divergence and relative α-entropy, with analysis of their properties and relevance.

Findings

The first measure satisfies the data-processing inequality and relates to hypothesis testing.

The second measure is useful in distributed task encoding.

Both measures reduce to Shannon's mutual information when α=1.

Abstract

Two families of dependence measures between random variables are introduced. They are based on the R\'enyi divergence of order $\alpha$ and the relative $\alpha$-entropy, respectively, and both dependence measures reduce to Shannon's mutual information when their order $\alpha$ is one. The first measure shares many properties with the mutual information, including the data-processing inequality, and can be related to the optimal error exponents in composite hypothesis testing. The second measure does not satisfy the data-processing inequality, but appears naturally in the context of distributed task encoding.