Open Problem: Order Optimal Regret Bounds for Kernel-Based Reinforcement Learning

TL;DR

This paper discusses the open problem of establishing order optimal regret bounds for kernel-based reinforcement learning, highlighting the gap in theoretical guarantees for non-linear function approximation methods.

Contribution

It reviews existing partial results and challenges related to deriving regret bounds for kernel-based RL, emphasizing the need for further theoretical development.

Findings

Identifies the gap in theoretical regret bounds for kernel-based RL.

Highlights the challenges in analyzing non-linear function approximation.

Reviews existing partial results and open problems.

Abstract

Reinforcement Learning (RL) has shown great empirical success in various application domains. The theoretical aspects of the problem have been extensively studied over past decades, particularly under tabular and linear Markov Decision Process structures. Recently, non-linear function approximation using kernel-based prediction has gained traction. This approach is particularly interesting as it naturally extends the linear structure, and helps explain the behavior of neural-network-based models at their infinite width limit. The analytical results however do not adequately address the performance guarantees for this case. We will highlight this open problem, overview existing partial results, and discuss related challenges.