RFFNet: Large-Scale Interpretable Kernel Methods via Random Fourier Features

TL;DR

RFFNet introduces a scalable kernel method that automatically learns feature relevances using random Fourier features and stochastic optimization, improving interpretability and efficiency for large datasets.

Contribution

It develops RFFNet, a novel large-scale kernel method with automatic relevance determination, enabling feature selection and interpretability in kernel learning.

Findings

Achieves low prediction error on real-world data.

Effectively identifies relevant features for interpretability.

Maintains small memory footprint and fast runtime.

Abstract

Kernel methods provide a flexible and theoretically grounded approach to nonlinear and nonparametric learning. While memory and run-time requirements hinder their applicability to large datasets, many low-rank kernel approximations, such as random Fourier features, were recently developed to scale up such kernel methods. However, these scalable approaches are based on approximations of isotropic kernels, which cannot remove the influence of irrelevant features. In this work, we design random Fourier features for a family of automatic relevance determination (ARD) kernels, and introduce RFFNet, a new large-scale kernel method that learns the kernel relevances' on the fly via first-order stochastic optimization. We present an effective initialization scheme for the method's non-convex objective function, evaluate if hard-thresholding RFFNet's learned relevances yield a sensible rule for variable selection, and perform an extensive ablation study of RFFNet's components. Numerical validation on simulated and real-world data shows that our approach has a small memory footprint and run-time, achieves low prediction error, and effectively identifies relevant features, thus leading to more interpretable solutions. We supply users with an efficient, PyTorch-based library, that adheres to the scikit-learn standard API and code for fully reproducing our results.