Statistical Learning Theory for Distributional Classification

TL;DR

This paper provides a theoretical analysis of kernel-based SVMs for distributional classification, establishing new learning guarantees and introducing novel technical tools for Gaussian kernels on Hilbert spaces.

Contribution

It offers the first oracle inequality and consistency results for distributional SVM classification, along with a new noise assumption and feature space construction.

Findings

Established a new oracle inequality for distributional SVMs.

Derived consistency and learning rates under specific noise conditions.

Introduced a novel feature space for Gaussian kernels on Hilbert spaces.

Abstract

In supervised learning with distributional inputs in the two-stage sampling setup, relevant to applications like learning-based medical screening or causal learning, the inputs (which are probability distributions) are not accessible in the learning phase, but only samples thereof. This problem is particularly amenable to kernel-based learning methods, where the distributions or samples are first embedded into a Hilbert space, often using kernel mean embeddings (KMEs), and then a standard kernel method like Support Vector Machines (SVMs) is applied, using a kernel defined on the embedding Hilbert space. In this work, we contribute to the theoretical analysis of this latter approach, with a particular focus on classification with distributional inputs using SVMs. We establish a new oracle inequality and derive consistency and learning rate results. Furthermore, for SVMs using the hinge loss and Gaussian kernels, we formulate a novel variant of an established noise assumption from the binary classification literature, under which we can establish learning rates. Finally, some of our technical tools like a new feature space for Gaussian kernels on Hilbert spaces are of independent interest.