Model-based Reinforcement Learning and the Eluder Dimension

TL;DR

This paper introduces regret bounds for model-based reinforcement learning that depend on the system's complexity, characterized by the Kolmogorov and Eluder dimensions, and proposes an efficient algorithm called PSRL.

Contribution

It provides the first unified regret bounds for model-based RL using the Eluder dimension and introduces a practical posterior sampling algorithm.

Findings

Regret bounds scale with the Eluder and Kolmogorov dimensions.

The proposed PSRL algorithm achieves these regret bounds.

State-of-the-art guarantees in several RL settings.

Abstract

We consider the problem of learning to optimize an unknown Markov decision process (MDP). We show that, if the MDP can be parameterized within some known function class, we can obtain regret bounds that scale with the dimensionality, rather than cardinality, of the system. We characterize this dependence explicitly as $\tilde{O}(\sqrt{d_K d_E T})$ where $T$ is time elapsed, $d_K$ is the Kolmogorov dimension and $d_E$ is the \emph{eluder dimension}. These represent the first unified regret bounds for model-based reinforcement learning and provide state of the art guarantees in several important settings. Moreover, we present a simple and computationally efficient algorithm \emph{posterior sampling for reinforcement learning} (PSRL) that satisfies these bounds.