Sigma-Delta and Distributed Noise-Shaping Quantization Methods for Random Fourier Features

TL;DR

This paper introduces low bit-depth Sigma-Delta and distributed noise-shaping quantization techniques for Random Fourier Features, enabling high-accuracy kernel approximations with minimal memory, and demonstrates their effectiveness on machine learning tasks.

Contribution

It presents novel quantization methods for RFFs that achieve polynomial and exponential decay in approximation error with respect to dimension and bits, respectively.

Findings

Quantized RFFs approximate kernels with high accuracy even at 1-bit quantization.

Error decay is polynomial with dimension and exponential with bits used.

Empirical results show superior performance compared to existing quantization methods.

Abstract

We propose the use of low bit-depth Sigma-Delta and distributed noise-shaping methods for quantizing the Random Fourier features (RFFs) associated with shift-invariant kernels. We prove that our quantized RFFs -- even in the case of $1$-bit quantization -- allow a high accuracy approximation of the underlying kernels, and the approximation error decays at least polynomially fast as the dimension of the RFFs increases. We also show that the quantized RFFs can be further compressed, yielding an excellent trade-off between memory use and accuracy. Namely, the approximation error now decays exponentially as a function of the bits used. Moreover, we empirically show by testing the performance of our methods on several machine learning tasks that our method compares favorably to other state of the art quantization methods in this context.