Provably Efficient Exploration in Constrained Reinforcement Learning:Posterior Sampling Is All You Need

TL;DR

This paper introduces a posterior sampling algorithm for constrained reinforcement learning in infinite-horizon settings, achieving near-optimal regret bounds and outperforming existing methods empirically.

Contribution

It provides the first Bayesian regret bounds for constrained Markov decision processes and demonstrates empirical superiority of the simple posterior sampling approach.

Findings

Achieves near-optimal Bayesian regret bounds of O(HS \u221a{AT})

Outperforms existing algorithms empirically in constrained RL tasks

Matches the lower bounds in the order of the time horizon T.

Abstract

We present a new algorithm based on posterior sampling for learning in constrained Markov decision processes (CMDP) in the infinite-horizon undiscounted setting. The algorithm achieves near-optimal regret bounds while being advantageous empirically compared to the existing algorithms. Our main theoretical result is a Bayesian regret bound for each cost component of \tilde{O} (HS \sqrt{AT}) for any communicating CMDP with S states, A actions, and bound on the hitting time H. This regret bound matches the lower bound in order of time horizon T and is the best-known regret bound for communicating CMDPs in the infinite-horizon undiscounted setting. Empirical results show that, despite its simplicity, our posterior sampling algorithm outperforms the existing algorithms for constrained reinforcement learning.