Projection-Based Constrained Policy Optimization

TL;DR

This paper introduces PCPO, a novel algorithm for learning control policies that optimize rewards while satisfying constraints, with theoretical guarantees and superior empirical performance on control tasks.

Contribution

The paper proposes PCPO, a new iterative algorithm combining reward optimization and constraint projection, with theoretical analysis and empirical validation.

Findings

PCPO reduces constraint violations by over 3.5 times compared to existing methods.

PCPO achieves approximately 15% higher reward in control tasks.

Theoretical bounds on reward improvement and constraint violation are established.

Abstract

We consider the problem of learning control policies that optimize a reward function while satisfying constraints due to considerations of safety, fairness, or other costs. We propose a new algorithm, Projection-Based Constrained Policy Optimization (PCPO). This is an iterative method for optimizing policies in a two-step process: the first step performs a local reward improvement update, while the second step reconciles any constraint violation by projecting the policy back onto the constraint set. We theoretically analyze PCPO and provide a lower bound on reward improvement, and an upper bound on constraint violation, for each policy update. We further characterize the convergence of PCPO based on two different metrics: $\normltwo$ norm and Kullback-Leibler divergence. Our empirical results over several control tasks demonstrate that PCPO achieves superior performance, averaging more than 3.5 times less constraint violation and around 15\% higher reward compared to state-of-the-art methods.