On the R\'{e}nyi Cross-Entropy

TL;DR

This paper explores the properties of Rényi cross-entropy, providing closed-form formulas for specific distribution classes and analyzing its behavior in Gaussian processes and Markov sources, enhancing understanding for deep learning applications.

Contribution

It derives new closed-form expressions for Rényi cross-entropy between distributions, including exponential family, Gaussian processes, and Markov sources, advancing theoretical understanding.

Findings

Closed-form formulas for Rényi cross-entropy with exponential family distributions

Analytical expression for cross-entropy rate in Gaussian processes

Formulas for finite-alphabet Markov sources

Abstract

The R\'{e}nyi cross-entropy measure between two distributions, a generalization of the Shannon cross-entropy, was recently used as a loss function for the improved design of deep learning generative adversarial networks. In this work, we examine the properties of this measure and derive closed-form expressions for it when one of the distributions is fixed and when both distributions belong to the exponential family. We also analytically determine a formula for the cross-entropy rate for stationary Gaussian processes and for finite-alphabet Markov sources.