A Simple Algorithm For Scaling Up Kernel Methods

TL;DR

This paper introduces a scalable random feature regression algorithm that enables kernel methods to handle large datasets efficiently, demonstrated on CIFAR-10, bridging the gap between neural networks and kernel approaches.

Contribution

A novel random feature regression algorithm that scales to very large numbers of features, making kernel methods more practical for large-scale applications.

Findings

Effective on CIFAR-10 dataset

Scales to virtually infinite random features

Bridges neural networks and kernel methods

Abstract

The recent discovery of the equivalence between infinitely wide neural networks (NNs) in the lazy training regime and Neural Tangent Kernels (NTKs) (Jacot et al., 2018) has revived interest in kernel methods. However, conventional wisdom suggests kernel methods are unsuitable for large samples due to their computational complexity and memory requirements. We introduce a novel random feature regression algorithm that allows us (when necessary) to scale to virtually infinite numbers of random features. We illustrate the performance of our method on the CIFAR-10 dataset.