Separated Proportional-Integral Lagrangian for Chance Constrained Reinforcement Learning

TL;DR

This paper introduces the separated proportional-integral Lagrangian (SPIL) algorithm for chance constrained reinforcement learning, enhancing safety and stability in real-world tasks like autonomous driving.

Contribution

The paper proposes a novel SPIL algorithm that combines proportional-integral control with chance constraints, addressing oscillations and constraint satisfaction issues in existing methods.

Findings

SPIL improves safety and performance in a car-following task.

SPIL ensures steady learning process and constraint satisfaction.

The method reduces conservatism through integral separation.

Abstract

Safety is essential for reinforcement learning (RL) applied in real-world tasks like autonomous driving. Chance constraints which guarantee the satisfaction of state constraints at a high probability are suitable to represent the requirements in real-world environment with uncertainty. Existing chance constrained RL methods like the penalty method and the Lagrangian method either exhibit periodic oscillations or cannot satisfy the constraints. In this paper, we address these shortcomings by proposing a separated proportional-integral Lagrangian (SPIL) algorithm. Taking a control perspective, we first interpret the penalty method and the Lagrangian method as proportional feedback and integral feedback control, respectively. Then, a proportional-integral Lagrangian method is proposed to steady learning process while improving safety. To prevent integral overshooting and reduce conservatism, we introduce the integral separation technique inspired by PID control. Finally, an analytical gradient of the chance constraint is utilized for model-based policy optimization. The effectiveness of SPIL is demonstrated by a narrow car-following task. Experiments indicate that compared with previous methods, SPIL improves the performance while guaranteeing safety, with a steady learning process.