Estimating R\'enyi's $\alpha$-Cross-Entropies in a Matrix-Based Way

TL;DR

This paper introduces unbiased, non-parametric, matrix-based estimators for Re9nyi's b5-cross-entropies using reproducing-kernel Hilbert spaces, enabling effective distribution comparison in high-dimensional settings.

Contribution

It proposes three novel measures of Re9nyi's b5-cross-entropies with unbiased, minimax-optimal estimation methods based on Gram matrices, satisfying all axioms of divergences.

Findings

Estimators are unbiased and minimax-optimal.

Convergence rate is independent of data dimensionality.

Suitable for high-dimensional distribution comparison.

Abstract

Conventional information-theoretic quantities assume access to probability distributions. Estimating such distributions is not trivial. Here, we consider function-based formulations of cross entropy that sidesteps this a priori estimation requirement. We propose three measures of R\'enyi's $\alpha$-cross-entropies in the setting of reproducing-kernel Hilbert spaces. Each measure has its appeals. We prove that we can estimate these measures in an unbiased, non-parametric, and minimax-optimal way. We do this via sample-constructed Gram matrices. This yields matrix-based estimators of R\'enyi's $\alpha$-cross-entropies. These estimators satisfy all of the axioms that R\'enyi established for divergences. Our cross-entropies can thus be used for assessing distributional differences. They are also appropriate for handling high-dimensional distributions, since the convergence rate of our estimator is independent of the sample dimensionality. Python code for implementing these measures can be found at https://github.com/isledge/MBRCE