Efficient KLMS and KRLS Algorithms: A Random Fourier Feature Perspective

TL;DR

This paper introduces a novel framework for online nonlinear modeling in RKHS using random Fourier features, enabling fixed-size solutions that are computationally efficient and maintain competitive convergence and accuracy.

Contribution

It proposes a new approach that maps data to a finite-dimensional Euclidean space with random features, avoiding sparsification and reducing computational complexity.

Findings

Algorithms are computationally more efficient than previous variants.

Solutions maintain fixed size, independent of iteration steps.

Convergence speed and error floors are comparable to existing methods.

Abstract

We present a new framework for online Least Squares algorithms for nonlinear modeling in RKH spaces (RKHS). Instead of implicitly mapping the data to a RKHS (e.g., kernel trick), we map the data to a finite dimensional Euclidean space, using random features of the kernel's Fourier transform. The advantage is that, the inner product of the mapped data approximates the kernel function. The resulting "linear" algorithm does not require any form of sparsification, since, in contrast to all existing algorithms, the solution's size remains fixed and does not increase with the iteration steps. As a result, the obtained algorithms are computationally significantly more efficient compared to previously derived variants, while, at the same time, they converge at similar speeds and to similar error floors.