Posterior Sampling for Continuing Environments

TL;DR

This paper introduces continuing PSRL, an extension of posterior sampling for reinforcement learning tailored for ongoing environments, providing theoretical regret bounds and formal analysis of the resampling exploration method.

Contribution

It develops and analyzes continuing PSRL, a novel approach that integrates posterior sampling into ongoing environments with rigorous regret bounds and exploration analysis.

Findings

Establishes an $ ilde{O}( au S oot{2}A T)$ Bayesian regret bound.

Provides the first formal analysis of resampling with randomized exploration.

Demonstrates the method's suitability for complex, continuing environments.

Abstract

We develop an extension of posterior sampling for reinforcement learning (PSRL) that is suited for a continuing agent-environment interface and integrates naturally into agent designs that scale to complex environments. The approach, continuing PSRL, maintains a statistically plausible model of the environment and follows a policy that maximizes expected $\gamma$-discounted return in that model. At each time, with probability $1-\gamma$, the model is replaced by a sample from the posterior distribution over environments. For a choice of discount factor that suitably depends on the horizon $T$, we establish an $\tilde{O}(\tau S \sqrt{A T})$ bound on the Bayesian regret, where $S$ is the number of environment states, $A$ is the number of actions, and $\tau$ denotes the reward averaging time, which is a bound on the duration required to accurately estimate the average reward of any policy. Our work is the first to formalize and rigorously analyze the resampling approach with randomized exploration.