Toward Large Kernel Models

TL;DR

This paper introduces EigenPro 3.0, a scalable kernel training algorithm that decouples model size from data size, enabling large-scale kernel models comparable to neural networks.

Contribution

The paper presents EigenPro 3.0, a novel algorithm that allows training large kernel models on big datasets by decoupling model size from data size.

Findings

EigenPro 3.0 scales to large datasets and models.

It outperforms existing kernel methods in scalability.

Demonstrates potential of large kernel models for big data.

Abstract

Recent studies indicate that kernel machines can often perform similarly or better than deep neural networks (DNNs) on small datasets. The interest in kernel machines has been additionally bolstered by the discovery of their equivalence to wide neural networks in certain regimes. However, a key feature of DNNs is their ability to scale the model size and training data size independently, whereas in traditional kernel machines model size is tied to data size. Because of this coupling, scaling kernel machines to large data has been computationally challenging. In this paper, we provide a way forward for constructing large-scale general kernel models, which are a generalization of kernel machines that decouples the model and data, allowing training on large datasets. Specifically, we introduce EigenPro 3.0, an algorithm based on projected dual preconditioned SGD and show scaling to model and data sizes which have not been possible with existing kernel methods.