Density Constrained Reinforcement Learning

TL;DR

This paper introduces a novel approach to constrained reinforcement learning by directly constraining state density functions, which simplifies constraint specification and guarantees constraint satisfaction.

Contribution

It proposes a new density-constrained RL framework, leveraging density-Q duality, with an algorithm that guarantees near-optimal solutions and constraint satisfaction.

Findings

Outperforms state-of-the-art CRL methods in various tasks

Guarantees constraint satisfaction with bounded error

Effective in safety-critical and resource-limited scenarios

Abstract

We study constrained reinforcement learning (CRL) from a novel perspective by setting constraints directly on state density functions, rather than the value functions considered by previous works. State density has a clear physical and mathematical interpretation, and is able to express a wide variety of constraints such as resource limits and safety requirements. Density constraints can also avoid the time-consuming process of designing and tuning cost functions required by value function-based constraints to encode system specifications. We leverage the duality between density functions and Q functions to develop an effective algorithm to solve the density constrained RL problem optimally and the constrains are guaranteed to be satisfied. We prove that the proposed algorithm converges to a near-optimal solution with a bounded error even when the policy update is imperfect. We use a set of comprehensive experiments to demonstrate the advantages of our approach over state-of-the-art CRL methods, with a wide range of density constrained tasks as well as standard CRL benchmarks such as Safety-Gym.