Reinforcement Learning with Convex Constraints

TL;DR

This paper introduces a flexible reinforcement learning framework that incorporates convex constraints on expected measurements, enabling safer, more diverse, and more expert-like behaviors with theoretical guarantees.

Contribution

It proposes a general algorithmic scheme for constrained RL that handles convex expected-value constraints, extending previous safety-focused methods to new properties like diversity.

Findings

Matches existing safety constraint algorithms in performance

Enforces new properties such as diversity

Applicable to model-free and model-based RL

Abstract

In standard reinforcement learning (RL), a learning agent seeks to optimize the overall reward. However, many key aspects of a desired behavior are more naturally expressed as constraints. For instance, the designer may want to limit the use of unsafe actions, increase the diversity of trajectories to enable exploration, or approximate expert trajectories when rewards are sparse. In this paper, we propose an algorithmic scheme that can handle a wide class of constraints in RL tasks: specifically, any constraints that require expected values of some vector measurements (such as the use of an action) to lie in a convex set. This captures previously studied constraints (such as safety and proximity to an expert), but also enables new classes of constraints (such as diversity). Our approach comes with rigorous theoretical guarantees and only relies on the ability to approximately solve standard RL tasks. As a result, it can be easily adapted to work with any model-free or model-based RL. In our experiments, we show that it matches previous algorithms that enforce safety via constraints, but can also enforce new properties that these algorithms do not incorporate, such as diversity.