Super-Samples from Kernel Herding

TL;DR

This paper introduces kernel herding, an extension of herding to continuous spaces using kernels, which deterministically generates samples that rapidly approximate a probability distribution with improved convergence rates.

Contribution

The paper presents a novel kernel herding algorithm that extends herding to continuous spaces and demonstrates its faster convergence in approximating distributions.

Findings

Kernel herding achieves an O(1/T) convergence rate.

Kernel herding effectively approximates Bayesian predictive distributions.

Deterministic sampling outperforms iid sampling in expectation approximation.

Abstract

We extend the herding algorithm to continuous spaces by using the kernel trick. The resulting "kernel herding" algorithm is an infinite memory deterministic process that learns to approximate a PDF with a collection of samples. We show that kernel herding decreases the error of expectations of functions in the Hilbert space at a rate O(1/T) which is much faster than the usual O(1/pT) for iid random samples. We illustrate kernel herding by approximating Bayesian predictive distributions.