Kernel-Based Function Approximation for Average Reward Reinforcement Learning: An Optimist No-Regret Algorithm

TL;DR

This paper introduces an optimistic kernel-based reinforcement learning algorithm for average reward settings, providing new theoretical guarantees and confidence intervals for improved performance analysis.

Contribution

It presents a novel no-regret algorithm with confidence bounds for kernel-based RL in the average reward setting, extending theoretical understanding.

Findings

Established no-regret guarantees for the algorithm

Derived a new confidence interval for kernel-based value prediction

Applicable to various reinforcement learning problems

Abstract

Reinforcement learning utilizing kernel ridge regression to predict the expected value function represents a powerful method with great representational capacity. This setting is a highly versatile framework amenable to analytical results. We consider kernel-based function approximation for RL in the infinite horizon average reward setting, also referred to as the undiscounted setting. We propose an optimistic algorithm, similar to acquisition function based algorithms in the special case of bandits. We establish novel no-regret performance guarantees for our algorithm, under kernel-based modelling assumptions. Additionally, we derive a novel confidence interval for the kernel-based prediction of the expected value function, applicable across various RL problems.