Online Attentive Kernel-Based Temporal Difference Learning

TL;DR

This paper introduces OAKTD, an online reinforcement learning algorithm that uses attentive kernel-based value function approximation with sparse representations, improving learning efficiency and stability in complex environments.

Contribution

The paper proposes a novel online RL method combining attentive kernel-based VFA with sparse representations and two-timescale optimization, with theoretical convergence guarantees.

Findings

OAKTD outperforms existing kernel-based TD algorithms.

OAKTD surpasses traditional TD with Tile Coding on benchmark tasks.

The method demonstrates improved data efficiency and stability.

Abstract

With rising uncertainty in the real world, online Reinforcement Learning (RL) has been receiving increasing attention due to its fast learning capability and improving data efficiency. However, online RL often suffers from complex Value Function Approximation (VFA) and catastrophic interference, creating difficulty for the deep neural network to be applied to an online RL algorithm in a fully online setting. Therefore, a simpler and more adaptive approach is introduced to evaluate value function with the kernel-based model. Sparse representations are superior at handling interference, indicating that competitive sparse representations should be learnable, non-prior, non-truncated and explicit when compared with current sparse representation methods. Moreover, in learning sparse representations, attention mechanisms are utilized to represent the degree of sparsification, and a smooth attentive function is introduced into the kernel-based VFA. In this paper, we propose an Online Attentive Kernel-Based Temporal Difference (OAKTD) algorithm using two-timescale optimization and provide convergence analysis of our proposed algorithm. Experimental evaluations showed that OAKTD outperformed several Online Kernel-based Temporal Difference (OKTD) learning algorithms in addition to the Temporal Difference (TD) learning algorithm with Tile Coding on public Mountain Car, Acrobot, CartPole and Puddle World tasks.