Supervised Kernel Thinning

TL;DR

This paper extends the kernel thinning algorithm to supervised learning, enabling faster training and inference for kernel methods like Nadaraya-Watson and kernel ridge regression, while maintaining statistical accuracy.

Contribution

It introduces Kernel-Thinned estimators for supervised learning, combining kernel thinning with classical methods to achieve quadratic speed-up and better statistical efficiency.

Findings

KT-based regressors are computationally more efficient than full-data estimators.

KT reduces training and inference times significantly.

KT provides improved statistical efficiency over i.i.d. subsampling.

Abstract

The kernel thinning algorithm of Dwivedi & Mackey (2024) provides a better-than-i.i.d. compression of a generic set of points. By generating high-fidelity coresets of size significantly smaller than the input points, KT is known to speed up unsupervised tasks like Monte Carlo integration, uncertainty quantification, and non-parametric hypothesis testing, with minimal loss in statistical accuracy. In this work, we generalize the KT algorithm to speed up supervised learning problems involving kernel methods. Specifically, we combine two classical algorithms--Nadaraya-Watson (NW) regression or kernel smoothing, and kernel ridge regression (KRR)--with KT to provide a quadratic speed-up in both training and inference times. We show how distribution compression with KT in each setting reduces to constructing an appropriate kernel, and introduce the Kernel-Thinned NW and Kernel-Thinned KRR estimators. We prove that KT-based regression estimators enjoy significantly superior computational efficiency over the full-data estimators and improved statistical efficiency over i.i.d. subsampling of the training data. En route, we also provide a novel multiplicative error guarantee for compressing with KT. We validate our design choices with both simulations and real data experiments.