Kernel Embeddings and the Separation of Measure Phenomenon

TL;DR

The paper proves that kernel covariance embeddings enable perfect separation of different continuous probability distributions, simplifying the distinction between measures in high-dimensional spaces.

Contribution

It establishes that testing measure equality is equivalent to testing Gaussian singularity in RKHS, revealing a fundamental separation phenomenon.

Findings

Kernel embeddings achieve perfect measure separation.

Singular Gaussian measures are supported on separate affine subspaces.

Small distribution perturbations are magnified in Gaussian embeddings.

Abstract

We prove that kernel covariance embeddings lead to information-theoretically perfect separation of distinct continuous probability distributions. In statistical terms, we establish that testing for the \emph{equality} of two non-atomic (Borel) probability measures on a locally compact uncountable Polish space is \emph{equivalent} to testing for the \emph{singularity} between two centered Gaussian measures on a reproducing kernel Hilbert space. The corresponding Gaussians are defined via the notion of kernel covariance embedding of a probability measure, and the Hilbert space is that generated by the embedding kernel. Distinguishing singular Gaussians is structurally simpler from an information-theoretic perspective than non-parametric two-sample testing, particularly in complex or high-dimensional domains. This is because singular Gaussians are supported on essentially separate and affine subspaces. Our proof leverages the classical Feldman-H\'{a}jek dichotomy, and shows that even a small perturbation of a continuous distribution will be maximally magnified through its Gaussian embedding. This ``separation of measure phenomenon'' appears to be a blessing of infinite dimensionality, by means of embedding, with the potential to inform the design of efficient inference tools in considerable generality. The elicitation of this phenomenon also appears to crystallize, in a precise and simple mathematical statement, a core mechanism underpinning the empirical effectiveness of kernel methods.