Simplex Random Features

TL;DR

Simplex Random Features (SimRFs) offer an unbiased, geometrically-coupled random feature method that minimizes mean square error for kernel approximation, outperforming previous methods in various machine learning tasks.

Contribution

Introduction of SimRFs, a novel random feature mechanism that achieves minimal MSE for kernel estimates and outperforms prior approaches like Orthogonal Random Features.

Findings

SimRFs achieve lower MSE in kernel approximation.

SimRFs outperform Orthogonal Random Features.

SimRFs show consistent improvements in classification and Transformers.

Abstract

We present Simplex Random Features (SimRFs), a new random feature (RF) mechanism for unbiased approximation of the softmax and Gaussian kernels by geometrical correlation of random projection vectors. We prove that SimRFs provide the smallest possible mean square error (MSE) on unbiased estimates of these kernels among the class of weight-independent geometrically-coupled positive random feature (PRF) mechanisms, substantially outperforming the previously most accurate Orthogonal Random Features at no observable extra cost. We present a more computationally expensive SimRFs+ variant, which we prove is asymptotically optimal in the broader family of weight-dependent geometrical coupling schemes (which permit correlations between random vector directions and norms). In extensive empirical studies, we show consistent gains provided by SimRFs in settings including pointwise kernel estimation, nonparametric classification and scalable Transformers.