The Fast Kernel Transform

TL;DR

The paper introduces the Fast Kernel Transform (FKT), an efficient algorithm for matrix-vector multiplications in kernel methods that is broadly applicable, accurate, and scalable to large datasets in moderate dimensions.

Contribution

The FKT leverages auto-differentiation and symbolic computation to efficiently compute kernel matrix-vector products for various kernels, improving scalability and accuracy over existing methods.

Findings

Achieves quasilinear complexity for kernel matrix-vector multiplications

Applicable to a wide range of kernels including Gaussian, Matern, and Green's functions

Demonstrates scalability and accuracy improvements in large-scale data applications

Abstract

Kernel methods are a highly effective and widely used collection of modern machine learning algorithms. A fundamental limitation of virtually all such methods are computations involving the kernel matrix that naively scale quadratically (e.g., constructing the kernel matrix and matrix-vector multiplication) or cubically (solving linear systems) with the size of the data set $N.$ We propose the Fast Kernel Transform (FKT), a general algorithm to compute matrix-vector multiplications (MVMs) for datasets in moderate dimensions with quasilinear complexity. Typically, analytically grounded fast multiplication methods require specialized development for specific kernels. In contrast, our scheme is based on auto-differentiation and automated symbolic computations that leverage the analytical structure of the underlying kernel. This allows the FKT to be easily applied to a broad class of kernels, including Gaussian, Matern, and Rational Quadratic covariance functions and physically motivated Green's functions, including those of the Laplace and Helmholtz equations. Furthermore, the FKT maintains a high, quantifiable, and controllable level of accuracy -- properties that many acceleration methods lack. We illustrate the efficacy and versatility of the FKT by providing timing and accuracy benchmarks and by applying it to scale the stochastic neighborhood embedding (t-SNE) and Gaussian processes to large real-world data sets.