Kernel methods through the roof: handling billions of points efficiently

TL;DR

This paper introduces a GPU-accelerated, out-of-core solver for kernel methods that efficiently handles datasets with billions of points, significantly improving scalability and speed.

Contribution

We develop a GPU-based preconditioned gradient solver for kernel methods that leverages parallelization, out-of-core computations, and optimized numerical precision for large-scale data.

Findings

Achieves dramatic speedups on datasets with billions of points

Maintains state-of-the-art performance in large-scale kernel learning

Provides an accessible software library for practical use

Abstract

Kernel methods provide an elegant and principled approach to nonparametric learning, but so far could hardly be used in large scale problems, since na\"ive implementations scale poorly with data size. Recent advances have shown the benefits of a number of algorithmic ideas, for example combining optimization, numerical linear algebra and random projections. Here, we push these efforts further to develop and test a solver that takes full advantage of GPU hardware. Towards this end, we designed a preconditioned gradient solver for kernel methods exploiting both GPU acceleration and parallelization with multiple GPUs, implementing out-of-core variants of common linear algebra operations to guarantee optimal hardware utilization. Further, we optimize the numerical precision of different operations and maximize efficiency of matrix-vector multiplications. As a result we can experimentally show dramatic speedups on datasets with billions of points, while still guaranteeing state of the art performance. Additionally, we make our software available as an easy to use library.