Reward Constrained Policy Optimization

TL;DR

This paper introduces RCPO, a novel multi-timescale method for constrained policy optimization in reinforcement learning, which guides policies to satisfy constraints using an alternative penalty signal, with proven convergence and empirical validation.

Contribution

It presents the first convergence-guaranteed multi-timescale approach for constrained policy optimization using an alternative penalty signal.

Findings

Proves convergence of RCPO algorithm.

Empirically demonstrates RCPO's ability to train constraint-satisfying policies.

Shows RCPO outperforms existing methods in constrained RL tasks.

Abstract

Solving tasks in Reinforcement Learning is no easy feat. As the goal of the agent is to maximize the accumulated reward, it often learns to exploit loopholes and misspecifications in the reward signal resulting in unwanted behavior. While constraints may solve this issue, there is no closed form solution for general constraints. In this work we present a novel multi-timescale approach for constrained policy optimization, called `Reward Constrained Policy Optimization' (RCPO), which uses an alternative penalty signal to guide the policy towards a constraint satisfying one. We prove the convergence of our approach and provide empirical evidence of its ability to train constraint satisfying policies.