Recurrent Memory for Online Interdomain Gaussian Processes

TL;DR

This paper introduces OHSVGP, an online Gaussian process model that effectively captures long-term dependencies in sequential data by integrating the HiPPO framework, leading to improved online prediction and continual learning.

Contribution

It presents a novel online GP model that incorporates HiPPO-based orthogonal projections for long-term memory, enabling efficient online updates and superior performance.

Findings

Outperforms existing online GP methods in predictive accuracy

Effectively preserves long-term memory in sequential data

Achieves computational efficiency through recurrence-based kernel updates

Abstract

We propose a novel online Gaussian process (GP) model that is capable of capturing long-term memory in sequential data in an online learning setting. Our model, Online HiPPO Sparse Variational Gaussian Process (OHSVGP), leverages the HiPPO (High-order Polynomial Projection Operators) framework, which is popularized in the RNN domain due to its long-range memory modeling capabilities. We interpret the HiPPO time-varying orthogonal projections as inducing variables with time-dependent orthogonal polynomial basis functions, which allows the SVGP inducing variables to memorize the process history. We show that the HiPPO framework fits naturally into the interdomain GP framework and demonstrate that the kernel matrices can also be updated online in a recurrence form based on the ODE evolution of HiPPO. We evaluate OHSVGP with online prediction for 1D time series, continual learning in discriminative GP model for data with multidimensional inputs, and deep generative modeling with sparse Gaussian process variational autoencoder, showing that it outperforms existing online GP methods in terms of predictive performance, long-term memory preservation, and computational efficiency.