Dynamic Online Ensembles of Basis Expansions

TL;DR

This paper introduces a flexible framework for online ensembling of basis expansion models, including Gaussian processes and polynomial regression, enhancing dynamic Bayesian learning with improved performance and model diversity.

Contribution

It generalizes random feature ensembling to various basis expansions and proposes a novel method to ensemble static and dynamic models together.

Findings

Ensembling different basis expansion models improves performance.

The method generalizes to Hilbert space Gaussian processes.

Ensembling static and dynamic models enhances adaptability.

Abstract

Practical Bayesian learning often requires (1) online inference, (2) dynamic models, and (3) ensembling over multiple different models. Recent advances have shown how to use random feature approximations to achieve scalable, online ensembling of Gaussian processes with desirable theoretical properties and fruitful applications. One key to these methods' success is the inclusion of a random walk on the model parameters, which makes models dynamic. We show that these methods can be generalized easily to any basis expansion model and that using alternative basis expansions, such as Hilbert space Gaussian processes, often results in better performance. To simplify the process of choosing a specific basis expansion, our method's generality also allows the ensembling of several entirely different models, for example, a Gaussian process and polynomial regression. Finally, we propose a novel method to ensemble static and dynamic models together.