All Random Features Representations are Equivalent

TL;DR

This paper proves that all random feature representations are fundamentally equivalent in approximation error when sampled optimally, simplifying the choice of representations in kernel methods.

Contribution

It derives an optimal sampling policy that equalizes and minimizes approximation error across all random feature representations.

Findings

All random feature representations have the same minimal approximation error under optimal sampling.

The optimal sampling policy achieves the lowest possible approximation error.

Practitioners can select any representation with confidence, given they sample optimally.

Abstract

Random features are a powerful technique for rewriting positive-definite kernels as linear products. They bring linear tools to bear in important nonlinear domains like KNNs and attention. Unfortunately, practical implementations require approximating an expectation, usually via sampling. This has led to the development of increasingly elaborate representations with ever lower sample error. We resolve this arms race by deriving an optimal sampling policy. Under this policy all random features representations have the same approximation error, which we show is the lowest possible. This means that we are free to choose whatever representation we please, provided we sample optimally.