FALKON: An Optimal Large Scale Kernel Method

TL;DR

FALKON is a scalable kernel method that efficiently handles millions of data points with optimal statistical accuracy, using novel algorithmic techniques to reduce computational and memory requirements.

Contribution

The paper introduces FALKON, a new kernel method algorithm that combines stochastic subsampling, iterative solvers, and preconditioning to enable large-scale applications.

Findings

FALKON achieves optimal statistical accuracy with linear memory use.

FALKON outperforms previous methods on large datasets.

FALKON operates efficiently on a single machine.

Abstract

Kernel methods provide a principled way to perform non linear, nonparametric learning. They rely on solid functional analytic foundations and enjoy optimal statistical properties. However, at least in their basic form, they have limited applicability in large scale scenarios because of stringent computational requirements in terms of time and especially memory. In this paper, we take a substantial step in scaling up kernel methods, proposing FALKON, a novel algorithm that allows to efficiently process millions of points. FALKON is derived combining several algorithmic principles, namely stochastic subsampling, iterative solvers and preconditioning. Our theoretical analysis shows that optimal statistical accuracy is achieved requiring essentially $O(n)$ memory and $O(n\sqrt{n})$ time. An extensive experimental analysis on large scale datasets shows that, even with a single machine, FALKON outperforms previous state of the art solutions, which exploit parallel/distributed architectures.