Model-based Chance-Constrained Reinforcement Learning via Separated Proportional-Integral Lagrangian

TL;DR

This paper introduces the separated proportional-integral Lagrangian (SPIL) algorithm for chance-constrained reinforcement learning, effectively reducing oscillations and conservatism in safety-critical applications like autonomous navigation.

Contribution

The paper proposes a novel SPIL algorithm that unifies proportional and integral control for safer, less conservative RL policies under uncertainty, with an integral separation technique and model-based gradient computation.

Findings

Reduces oscillations in RL safety policies.

Decreases conservatism in policy learning.

Successfully applied to real-world robot navigation.

Abstract

Safety is essential for reinforcement learning (RL) applied in the real world. Adding chance constraints (or probabilistic constraints) is a suitable way to enhance RL safety under uncertainty. Existing chance-constrained RL methods like the penalty methods and the Lagrangian methods either exhibit periodic oscillations or learn an over-conservative or unsafe policy. In this paper, we address these shortcomings by proposing a separated proportional-integral Lagrangian (SPIL) algorithm. We first review the constrained policy optimization process from a feedback control perspective, which regards the penalty weight as the control input and the safe probability as the control output. Based on this, the penalty method is formulated as a proportional controller, and the Lagrangian method is formulated as an integral controller. We then unify them and present a proportional-integral Lagrangian method to get both their merits, with an integral separation technique to limit the integral value in a reasonable range. To accelerate training, the gradient of safe probability is computed in a model-based manner. We demonstrate our method can reduce the oscillations and conservatism of RL policy in a car-following simulation. To prove its practicality, we also apply our method to a real-world mobile robot navigation task, where our robot successfully avoids a moving obstacle with highly uncertain or even aggressive behaviors.