Robust and Adaptive Temporal-Difference Learning Using An Ensemble of Gaussian Processes

TL;DR

This paper introduces scalable, robust, and adaptive Gaussian process-based temporal-difference learning methods for policy evaluation in reinforcement learning, capable of handling adversarial settings and selecting kernels dynamically.

Contribution

It develops the OS-GPTD and OS-EGPTD algorithms that improve value function estimation with online scalability, robustness, and kernel ensemble adaptation.

Findings

OS-GPTD performs well in large state spaces.

OS-EGPTD adaptively selects kernels for better accuracy.

Both methods outperform fixed kernel approaches in benchmarks.

Abstract

Value function approximation is a crucial module for policy evaluation in reinforcement learning when the state space is large or continuous. The present paper takes a generative perspective on policy evaluation via temporal-difference (TD) learning, where a Gaussian process (GP) prior is presumed on the sought value function, and instantaneous rewards are probabilistically generated based on value function evaluations at two consecutive states. Capitalizing on a random feature-based approximant of the GP prior, an online scalable (OS) approach, termed {OS-GPTD}, is developed to estimate the value function for a given policy by observing a sequence of state-reward pairs. To benchmark the performance of OS-GPTD even in an adversarial setting, where the modeling assumptions are violated, complementary worst-case analyses are performed by upper-bounding the cumulative Bellman error as well as the long-term reward prediction error, relative to their counterparts from a fixed value function estimator with the entire state-reward trajectory in hindsight. Moreover, to alleviate the limited expressiveness associated with a single fixed kernel, a weighted ensemble (E) of GP priors is employed to yield an alternative scheme, termed OS-EGPTD, that can jointly infer the value function, and select interactively the EGP kernel on-the-fly. Finally, performances of the novel OS-(E)GPTD schemes are evaluated on two benchmark problems.