Dynamic Regret for Online Regression in RKHS via Discounted VAW and Subspace Approximation

TL;DR

This paper develops a method for online regression in RKHS that achieves dynamic regret bounds by combining subspace approximation, ensemble forecasting, and spectral truncation techniques, applicable to various kernels.

Contribution

It introduces a general orthogonal truncation approach for finite-dimensional approximation in RKHS, extending discounted VAW methods with new theoretical guarantees.

Findings

Fast-regime bounds for Gaussian and analytic kernels

Spectral approximation bounds depending on eigenvalue decay

Application to Matérn kernels with subspace constructions

Abstract

We study online regression with the square loss in a reproducing kernel Hilbert space under a dynamic regret criterion. The learner is compared with a time-varying comparator sequence, and the bounds depend on its path length in the RKHS norm. The proposed method transfers the finite-dimensional discounted Vovk--Azoury--Warmuth approach of Jacobsen \& Cutkosky (2024) to the RKHS setting by means of finite-dimensional subspace approximations. For a fixed subspace, we run a VAW-based ensemble of discounted VAW forecasters over a geometric grid of discount factors. The additional approximation error is controlled by the uniform projection error of kernel sections. We then introduce a general orthogonal truncation method: starting from a feature expansion of the kernel, we construct the associated RKHS by introducing an inner product that makes the feature functions orthonormal, and then use the spans of the first basis functions as finite-dimensional approximation spaces. The resulting subspace reduction is applied to several approximation schemes. Explicit feature expansions yield fast-regime bounds for Gaussian and analytic dot-product kernels. Mercer truncations provide a spectral approximation method and lead to dynamic regret bounds in fast and slow regimes, depending on the eigenvalue decay. Finally, we study subspaces spanned by kernel sections and apply this construction to Mat\'ern kernels.