Kullback-Leibler and Renyi divergences in reproducing kernel Hilbert space and Gaussian process settings

TL;DR

This paper develops regularized Kullback-Leibler and Rènyi divergences in RKHS and Gaussian process contexts, enabling consistent, dimension-independent estimation from finite data using Hilbert-Schmidt operators and Log-Det divergences.

Contribution

It introduces novel formulations of divergences between probability measures and Gaussian processes in RKHS and Hilbert space settings, with proven continuity and efficient finite-sample estimation methods.

Findings

Divergences are continuous in Hilbert-Schmidt norm.

Finite-dimensional estimators are consistent and dimension-independent.

Numerical experiments validate the theoretical formulations.

Abstract

In this work, we present formulations for regularized Kullback-Leibler and R\'enyi divergences via the Alpha Log-Determinant (Log-Det) divergences between positive Hilbert-Schmidt operators on Hilbert spaces in two different settings, namely (i) covariance operators and Gaussian measures defined on reproducing kernel Hilbert spaces (RKHS); and (ii) Gaussian processes with squared integrable sample paths. For characteristic kernels, the first setting leads to divergences between arbitrary Borel probability measures on a complete, separable metric space. We show that the Alpha Log-Det divergences are continuous in the Hilbert-Schmidt norm, which enables us to apply laws of large numbers for Hilbert space-valued random variables. As a consequence of this, we show that, in both settings, the infinite-dimensional divergences can be consistently and efficiently estimated from their finite-dimensional versions, using finite-dimensional Gram matrices/Gaussian measures and finite sample data, with {\it dimension-independent} sample complexities in all cases. RKHS methodology plays a central role in the theoretical analysis in both settings. The mathematical formulation is illustrated by numerical experiments.