On the Error of Random Fourier Features

TL;DR

This paper improves the theoretical understanding of random Fourier features used for kernel approximation, providing tighter error bounds and insights into variance, with implications for scalable machine learning methods.

Contribution

It offers improved uniform error bounds and novel insights into variance and approximation error of random Fourier features, challenging assumptions about their variants.

Findings

Improved uniform error bounds for random Fourier features.

The more widely used variant has higher variance and worse bounds for Gaussian kernels.

Provides insights into the embedding's variance and approximation error.

Abstract

Kernel methods give powerful, flexible, and theoretically grounded approaches to solving many problems in machine learning. The standard approach, however, requires pairwise evaluations of a kernel function, which can lead to scalability issues for very large datasets. Rahimi and Recht (2007) suggested a popular approach to handling this problem, known as random Fourier features. The quality of this approximation, however, is not well understood. We improve the uniform error bound of that paper, as well as giving novel understandings of the embedding's variance, approximation error, and use in some machine learning methods. We also point out that surprisingly, of the two main variants of those features, the more widely used is strictly higher-variance for the Gaussian kernel and has worse bounds.