Relative $\alpha$-Entropy Minimizers Subject to Linear Statistical Constraints

TL;DR

This paper investigates the minimization of a family of relative entropies called relative α-entropies under linear constraints, revealing that the optimal distributions follow a power-law form and generalize classical entropy principles.

Contribution

It introduces and analyzes the properties of relative α-entropies, extending the classical relative entropy framework to a parametric family with applications in statistical inference.

Findings

Minimizers exhibit power-law distributions.

Relative α-entropies satisfy the Pythagorean property.

The approach generalizes maximum Rényi or Tsallis entropy principles.

Abstract

We study minimization of a parametric family of relative entropies, termed relative $\alpha$-entropies (denoted $\mathscr{I}_{\alpha}(P,Q)$). These arise as redundancies under mismatched compression when cumulants of compressed lengths are considered instead of expected compressed lengths. These parametric relative entropies are a generalization of the usual relative entropy (Kullback-Leibler divergence). Just like relative entropy, these relative $\alpha$-entropies behave like squared Euclidean distance and satisfy the Pythagorean property. Minimization of $\mathscr{I}_{\alpha}(P,Q)$ over the first argument on a set of probability distributions that constitutes a linear family is studied. Such a minimization generalizes the maximum R\'{e}nyi or Tsallis entropy principle. The minimizing probability distribution (termed $\mathscr{I}_{\alpha}$-projection) for a linear family is shown to have a power-law.