Learning in High-Dimensional Feature Spaces Using ANOVA-Based Fast Matrix-Vector Multiplication

TL;DR

This paper introduces an efficient method for computing kernel matrix-vector products in high-dimensional spaces using ANOVA kernels and NFFT, enabling scalable kernel ridge regression.

Contribution

It presents a novel approach combining ANOVA kernels with NFFT for fast matrix-vector multiplication in high-dimensional learning tasks.

Findings

Achieves linear complexity for fixed accuracy in matrix-vector products.

Demonstrates improved scalability in kernel ridge regression.

Validates effectiveness on multiple datasets.

Abstract

Kernel matrices are crucial in many learning tasks such as support vector machines or kernel ridge regression. The kernel matrix is typically dense and large-scale. Depending on the dimension of the feature space even the computation of all of its entries in reasonable time becomes a challenging task. For such dense matrices the cost of a matrix-vector product scales quadratically with the dimensionality N , if no customized methods are applied. We propose the use of an ANOVA kernel, where we construct several kernels based on lower-dimensional feature spaces for which we provide fast algorithms realizing the matrix-vector products. We employ the non-equispaced fast Fourier transform (NFFT), which is of linear complexity for fixed accuracy. Based on a feature grouping approach, we then show how the fast matrix-vector products can be embedded into a learning method choosing kernel ridge regression and the conjugate gradient solver. We illustrate the performance of our approach on several data sets.