Posterior Sampling for Large Scale Reinforcement Learning

TL;DR

This paper introduces DS-PSRL, a practical and efficient non-episodic posterior sampling algorithm for large-scale reinforcement learning, with proven regret bounds and broad applicability.

Contribution

It presents a deterministic schedule PSRL algorithm that improves efficiency and generality over existing methods, with theoretical guarantees.

Findings

Outperforms state-of-the-art PSRL algorithms on benchmark problems

Provides a Bayesian regret bound under mild assumptions

Applicable to multi-parameter and continuous state-action problems

Abstract

We propose a practical non-episodic PSRL algorithm that unlike recent state-of-the-art PSRL algorithms uses a deterministic, model-independent episode switching schedule. Our algorithm termed deterministic schedule PSRL (DS-PSRL) is efficient in terms of time, sample, and space complexity. We prove a Bayesian regret bound under mild assumptions. Our result is more generally applicable to multiple parameters and continuous state action problems. We compare our algorithm with state-of-the-art PSRL algorithms on standard discrete and continuous problems from the literature. Finally, we show how the assumptions of our algorithm satisfy a sensible parametrization for a large class of problems in sequential recommendations.