Adaptive Kernel Methods

TL;DR

This paper introduces adaptive kernel methods with learnable parameters that depend on the dataset and loss function, enabling efficient large-scale nonlinear modeling beyond traditional fixed RKHS approaches.

Contribution

It proposes a new class of adaptive kernel methods with parameter-dependent solution spaces, improving efficiency and scalability for large datasets.

Findings

Efficient approximation of kernels for infinite-dimensional RKHSs.

Construction of fixed-dimensional, parameter-dependent solution spaces.

Demonstrated effectiveness through numerical experiments.

Abstract

Kernel methods approximate nonlinear maps in a data-driven manner by projecting the target map onto a finite-dimensional Hilbert space called the solution space. Traditionally, this space is a subspace of a fixed ambient reproducing kernel Hilbert space (RKHS), determined solely by the chosen kernel and the dataset, whose elements identify the basis elements. Consequently, the projection operator underlying the kernel method depends on the loss function, the dataset, and the choice of ambient RKHS. In this study, we consider kernel methods whose solution spaces also depend on learnable parameters that are independent of the dataset. The resulting methods can be viewed as variable projection operators that depend on the loss function, the dataset, and the new learnable parameters instead of a fixed RKHS. This work has two main contributions. First, we propose an efficient approximation of kernels associated with infinite-dimensional RKHSs, commonly used to reduce the solution-space dimension for large datasets. Second, we construct fixed-dimensional, parameter-dependent solution spaces that enable highly efficient kernel models suitable for large-scale problems without the need to approximate kernels of infinite-dimensional RKHSs. Our novel family of adaptive kernel methods generalizes earlier approaches, including Random Fourier Features, and we demonstrate their effectiveness through several numerical experiments.