Local Random Feature Approximations of the Gaussian Kernel

TL;DR

This paper investigates random feature approximations for the Gaussian kernel, identifies limitations with high-frequency data, and introduces a localization scheme that enhances approximation quality and regression performance.

Contribution

It introduces a novel localization scheme for random feature approximations that improves Gaussian kernel modeling, especially for high-frequency data.

Findings

Localization scheme significantly improves approximation accuracy.

Enhanced performance in Gaussian process regression tasks.

Method effective across various data sizes and dimensions.

Abstract

A fundamental drawback of kernel-based statistical models is their limited scalability to large data sets, which requires resorting to approximations. In this work, we focus on the popular Gaussian kernel and on techniques to linearize kernel-based models by means of random feature approximations. In particular, we do so by studying a less explored random feature approximation based on Maclaurin expansions and polynomial sketches. We show that such approaches yield poor results when modelling high-frequency data, and we propose a novel localization scheme that improves kernel approximations and downstream performance significantly in this regime. We demonstrate these gains on a number of experiments involving the application of Gaussian process regression to synthetic and real-world data of different data sizes and dimensions.