On Sampling Random Features From Empirical Leverage Scores: Implementation and Theoretical Guarantees

TL;DR

This paper investigates empirical leverage score-based sampling for random features in kernel approximation, providing theoretical guarantees and demonstrating improved performance over traditional methods through experiments.

Contribution

It introduces a practical approach for data-dependent sampling of random features using empirical leverage scores, with theoretical performance bounds and empirical validation.

Findings

Empirical leverage score sampling outperforms Monte Carlo sampling in experiments.

The method is competitive with supervised kernel learning without using label information.

Theoretical bounds reveal a trade-off between kernel approximation and eigenvalue decay.

Abstract

Random features provide a practical framework for large-scale kernel approximation and supervised learning. It has been shown that data-dependent sampling of random features using leverage scores can significantly reduce the number of features required to achieve optimal learning bounds. Leverage scores introduce an optimized distribution for features based on an infinite-dimensional integral operator (depending on input distribution), which is impractical to sample from. Focusing on empirical leverage scores in this paper, we establish an out-of-sample performance bound, revealing an interesting trade-off between the approximated kernel and the eigenvalue decay of another kernel in the domain of random features defined based on data distribution. Our experiments verify that the empirical algorithm consistently outperforms vanilla Monte Carlo sampling, and with a minor modification the method is even competitive to supervised data-dependent kernel learning, without using the output (label) information.