Learning to Embed Distributions via Maximum Kernel Entropy

TL;DR

This paper introduces a novel unsupervised method for learning data-dependent kernels for distribution classification by maximizing entropy in the space of probability measure embeddings, improving downstream discriminative tasks.

Contribution

It proposes a new entropy maximization objective for learning kernels directly from data, addressing the challenge of kernel selection in distribution regression.

Findings

The learned kernel has a favorable geometric structure for classification.

The method performs well across different data modalities.

Theoretical analysis supports the effectiveness of the embedding space.

Abstract

Empirical data can often be considered as samples from a set of probability distributions. Kernel methods have emerged as a natural approach for learning to classify these distributions. Although numerous kernels between distributions have been proposed, applying kernel methods to distribution regression tasks remains challenging, primarily because selecting a suitable kernel is not straightforward. Surprisingly, the question of learning a data-dependent distribution kernel has received little attention. In this paper, we propose a novel objective for the unsupervised learning of data-dependent distribution kernel, based on the principle of entropy maximization in the space of probability measure embeddings. We examine the theoretical properties of the latent embedding space induced by our objective, demonstrating that its geometric structure is well-suited for solving downstream discriminative tasks. Finally, we demonstrate the performance of the learned kernel across different modalities.