On the Relationship between Online Gaussian Process Regression and Kernel Least Mean Squares Algorithms

TL;DR

This paper explores the connection between online Gaussian process regression and kernel least mean squares algorithms, revealing how KLMS approximates a fixed posterior covariance and explaining their uncertainty handling and performance differences.

Contribution

It uncovers the probabilistic relationship between GP regression and KLMS, providing insights into their uncertainty management and linking specific KLMS algorithms to particular models.

Findings

KLMS approximates a fixed posterior covariance in GP regression.

Different KLMS algorithms correspond to specific parametric models.

Understanding this relationship explains performance differences among algorithms.

Abstract

We study the relationship between online Gaussian process (GP) regression and kernel least mean squares (KLMS) algorithms. While the latter have no capacity of storing the entire posterior distribution during online learning, we discover that their operation corresponds to the assumption of a fixed posterior covariance that follows a simple parametric model. Interestingly, several well-known KLMS algorithms correspond to specific cases of this model. The probabilistic perspective allows us to understand how each of them handles uncertainty, which could explain some of their performance differences.