Consistent Interpolating Ensembles via the Manifold-Hilbert Kernel

TL;DR

This paper introduces a novel ensemble classification method that interpolates training data and guarantees consistency across various data distributions by leveraging the manifold-Hilbert kernel on Riemannian manifolds.

Contribution

It defines the manifold-Hilbert kernel for data on Riemannian manifolds and proves its weak consistency for kernel smoothing regression, extending overparametrized learning theory to ensemble methods.

Findings

Kernel smoothing with the manifold-Hilbert kernel is weakly consistent.

On the sphere, the kernel can be realized as an infinite ensemble of partition-based classifiers.

The method interpolates training data while maintaining generalization guarantees.

Abstract

Recent research in the theory of overparametrized learning has sought to establish generalization guarantees in the interpolating regime. Such results have been established for a few common classes of methods, but so far not for ensemble methods. We devise an ensemble classification method that simultaneously interpolates the training data, and is consistent for a broad class of data distributions. To this end, we define the manifold-Hilbert kernel for data distributed on a Riemannian manifold. We prove that kernel smoothing regression using the manifold-Hilbert kernel is weakly consistent in the setting of Devroye et al. 1998. For the sphere, we show that the manifold-Hilbert kernel can be realized as a weighted random partition kernel, which arises as an infinite ensemble of partition-based classifiers.