Reinforcement Learning in Feature Space: Matrix Bandit, Kernels, and Regret Bound

TL;DR

This paper introduces MatrixRL, an online reinforcement learning algorithm that learns low-dimensional representations of transition models using features or kernels, achieving near-optimal regret bounds in high-dimensional settings.

Contribution

The paper presents the first near-optimal regret bounds for RL with feature and kernel representations, extending theoretical guarantees to high-dimensional and kernelized models.

Findings

MatrixRL achieves regret bound O(H^2 d log T √T) with features.

Kernelized MatrixRL achieves regret bound O(H^2 ˜d log T √T) with kernels.

First regret bounds for feature and kernel-based RL that are near-optimal in T and dimension.

Abstract

Exploration in reinforcement learning (RL) suffers from the curse of dimensionality when the state-action space is large. A common practice is to parameterize the high-dimensional value and policy functions using given features. However existing methods either have no theoretical guarantee or suffer a regret that is exponential in the planning horizon $H$. In this paper, we propose an online RL algorithm, namely the MatrixRL, that leverages ideas from linear bandit to learn a low-dimensional representation of the probability transition model while carefully balancing the exploitation-exploration tradeoff. We show that MatrixRL achieves a regret bound ${O}\big(H^2d\log T\sqrt{T}\big)$ where $d$ is the number of features. MatrixRL has an equivalent kernelized version, which is able to work with an arbitrary kernel Hilbert space without using explicit features. In this case, the kernelized MatrixRL satisfies a regret bound ${O}\big(H^2\widetilde{d}\log T\sqrt{T}\big)$, where $\widetilde{d}$ is the effective dimension of the kernel space. To our best knowledge, for RL using features or kernels, our results are the first regret bounds that are near-optimal in time $T$ and dimension $d$ (or $\widetilde{d}$) and polynomial in the planning horizon $H$.