A Simple Reward-free Approach to Constrained Reinforcement Learning

TL;DR

This paper introduces a straightforward method that leverages reward-free reinforcement learning to efficiently solve constrained RL problems, including in tabular and linear function approximation settings, with minimal additional complexity.

Contribution

It presents a simple meta-algorithm that uses reward-free RL oracles to address constrained RL, providing sharp sample complexity bounds and extending to Markov games.

Findings

Achieves near-optimal sample complexity for constrained RL in tabular MDPs.

Extends the approach to tabular two-player Markov games.

Provides new results for constrained RL with linear function approximation.

Abstract

In constrained reinforcement learning (RL), a learning agent seeks to not only optimize the overall reward but also satisfy the additional safety, diversity, or budget constraints. Consequently, existing constrained RL solutions require several new algorithmic ingredients that are notably different from standard RL. On the other hand, reward-free RL is independently developed in the unconstrained literature, which learns the transition dynamics without using the reward information, and thus naturally capable of addressing RL with multiple objectives under the common dynamics. This paper bridges reward-free RL and constrained RL. Particularly, we propose a simple meta-algorithm such that given any reward-free RL oracle, the approachability and constrained RL problems can be directly solved with negligible overheads in sample complexity. Utilizing the existing reward-free RL solvers, our framework provides sharp sample complexity results for constrained RL in the tabular MDP setting, matching the best existing results up to a factor of horizon dependence; our framework directly extends to a setting of tabular two-player Markov games, and gives a new result for constrained RL with linear function approximation.