Minimization Problems Based on Relative $\alpha$-Entropy I: Forward Projection

TL;DR

This paper introduces and analyzes a family of generalized relative entropies called relative α-entropies, exploring their minimization properties, geometric behavior, and implications for statistical inference, extending classical divergence concepts.

Contribution

The paper defines relative α-entropies, proves existence of minimizers on convex sets, and characterizes forward projections as power-law distributions, generalizing classical divergence and entropy principles.

Findings

Existence of minimizers of relative α-entropies on convex sets.

Forward α-projections obey power-law distributions.

Relative α-entropies satisfy the Pythagorean property.

Abstract

Minimization problems with respect to a one-parameter family of generalized relative entropies are studied. These relative entropies, which we term relative $\alpha$-entropies (denoted $\mathscr{I}_{\alpha}$), arise as redundancies under mismatched compression when cumulants of compressed lengths are considered instead of expected compressed lengths. These parametric relative entropies are a generalization of the usual relative entropy (Kullback-Leibler divergence). Just like relative entropy, these relative $\alpha$-entropies behave like squared Euclidean distance and satisfy the Pythagorean property. Minimizers of these relative $\alpha$-entropies on closed and convex sets are shown to exist. Such minimizations generalize the maximum R\'{e}nyi or Tsallis entropy principle. The minimizing probability distribution (termed forward $\mathscr{I}_{\alpha}$-projection) for a linear family is shown to obey a power-law. Other results in connection with statistical inference, namely subspace transitivity and iterated projections, are also established. In a companion paper, a related minimization problem of interest in robust statistics that leads to a reverse $\mathscr{I}_{\alpha}$-projection is studied.