Projection Theorems for the R\'enyi Divergence on $\alpha$-Convex Sets

TL;DR

This paper develops projection theorems for Rényi divergence on $oldsymbol{ extit{ extalpha}}$-convex sets, introducing $oldsymbol{ extalpha}}$-exponential families and establishing orthogonality, with implications for statistical physics and robust statistics.

Contribution

It provides new existence conditions for forward projections, generalizes exponential families to $oldsymbol{ extalpha}}$-exponential families, and establishes orthogonality and convergence results for these projections.

Findings

Proved a sufficient condition for the existence of forward projections.

Introduced $oldsymbol{ extalpha}}$-exponential families as an extension of exponential families.

Established an orthogonality relationship enabling conversion between forward and reverse projections.

Abstract

This paper studies forward and reverse projections for the R\'{e}nyi divergence of order $\alpha \in (0, \infty)$ on $\alpha$-convex sets. The forward projection on such a set is motivated by some works of Tsallis {\em et al.} in statistical physics, and the reverse projection is motivated by robust statistics. In a recent work, van Erven and Harremo\"es proved a Pythagorean inequality for R\'{e}nyi divergences on $\alpha$-convex sets under the assumption that the forward projection exists. Continuing this study, a sufficient condition for the existence of forward projection is proved for probability measures on a general alphabet. For $\alpha \in (1, \infty)$, the proof relies on a new Apollonius theorem for the Hellinger divergence, and for $\alpha \in (0,1)$, the proof relies on the Banach-Alaoglu theorem from functional analysis. Further projection results are then obtained in the finite alphabet setting. These include a projection theorem on a specific $\alpha$-convex set, which is termed an {\em $\alpha$-linear family}, generalizing a result by Csisz\'ar for $\alpha \neq 1$. The solution to this problem yields a parametric family of probability measures which turns out to be an extension of the exponential family, and it is termed an {\em $\alpha$-exponential family}. An orthogonality relationship between the $\alpha$-exponential and $\alpha$-linear families is established, and it is used to turn the reverse projection on an $\alpha$-exponential family into a forward projection on a $\alpha$-linear family. This paper also proves a convergence result of an iterative procedure used to calculate the forward projection on an intersection of a finite number of $\alpha$-linear families.