Constrained Upper Confidence Reinforcement Learning

TL;DR

This paper introduces C-UCRL, an algorithm for constrained reinforcement learning with known transition dynamics but unknown rewards and constraints, achieving sub-linear regret while ensuring safety constraints are met.

Contribution

It extends upper confidence reinforcement learning to constrained settings with known transitions, providing a regret guarantee and safety compliance.

Findings

C-UCRL achieves sub-linear regret of order $O(T^{3/4}\sqrt{\log(T/\delta)})$

The algorithm guarantees constraint satisfaction with high probability

Illustrative examples demonstrate practical effectiveness

Abstract

Constrained Markov Decision Processes are a class of stochastic decision problems in which the decision maker must select a policy that satisfies auxiliary cost constraints. This paper extends upper confidence reinforcement learning for settings in which the reward function and the constraints, described by cost functions, are unknown a priori but the transition kernel is known. Such a setting is well-motivated by a number of applications including exploration of unknown, potentially unsafe, environments. We present an algorithm C-UCRL and show that it achieves sub-linear regret ($ O(T^{\frac{3}{4}}\sqrt{\log(T/\delta)})$) with respect to the reward while satisfying the constraints even while learning with probability $1-\delta$. Illustrative examples are provided.