Learning Bounds for Greedy Approximation with Explicit Feature Maps from Multiple Kernels

TL;DR

This paper introduces a greedy method for selecting explicit features from multiple kernels to improve nonlinear kernel approximation, providing theoretical error bounds and demonstrating efficiency and accuracy gains over existing methods.

Contribution

It proposes a novel greedy feature selection approach from multiple kernels with theoretical error bounds, enhancing kernel approximation efficiency and accuracy.

Findings

Achieves lower test error with fewer explicit features.

Provides theoretical bounds linking approximation and spectral errors.

Outperforms state-of-the-art data-dependent random features in experiments.

Abstract

Nonlinear kernels can be approximated using finite-dimensional feature maps for efficient risk minimization. Due to the inherent trade-off between the dimension of the (mapped) feature space and the approximation accuracy, the key problem is to identify promising (explicit) features leading to a satisfactory out-of-sample performance. In this work, we tackle this problem by efficiently choosing such features from multiple kernels in a greedy fashion. Our method sequentially selects these explicit features from a set of candidate features using a correlation metric. We establish an out-of-sample error bound capturing the trade-off between the error in terms of explicit features (approximation error) and the error due to spectral properties of the best model in the Hilbert space associated to the combined kernel (spectral error). The result verifies that when the (best) underlying data model is sparse enough, i.e., the spectral error is negligible, one can control the test error with a small number of explicit features, that can scale poly-logarithmically with data. Our empirical results show that given a fixed number of explicit features, the method can achieve a lower test error with a smaller time cost, compared to the state-of-the-art in data-dependent random features.