Enforcing Cooperative Safety for Reinforcement Learning-based Mixed-Autonomy Platoon Control

TL;DR

This paper introduces a novel safe MARL framework for mixed-autonomy platoons that ensures cooperative safety of CAVs and HDVs by integrating control barrier functions and uncertainty estimation, improving safety guarantees.

Contribution

It proposes a cooperative safety framework using control barrier functions and conformal prediction within MARL, providing safety guarantees and handling unknown vehicle behaviors.

Findings

Enhanced system-level safety in mixed-autonomy platoons

Effective safety guarantees via CBF and QP integration

Accurate behavior estimation with uncertainty qualification

Abstract

It is recognized that the control of mixed-autonomy platoons comprising connected and automated vehicles (CAVs) and human-driven vehicles (HDVs) can enhance traffic flow. Among existing methods, Multi-Agent Reinforcement Learning (MARL) appears to be a promising control strategy because it can manage complex scenarios in real time. However, current research on MARL-based mixed-autonomy platoon control suffers from several limitations. First, existing MARL approaches address safety by penalizing safety violations in the reward function, thus lacking theoretical safety guarantees due to the black-box nature of RL. Second, few studies have explored the cooperative safety of multi-CAV platoons, where CAVs can be coordinated to further enhance the system-level safety involving the safety of both CAVs and HDVs. Third, existing work tends to make an unrealistic assumption that the behavior of HDVs and CAVs is publicly known and rationale. To bridge the research gaps, we propose a safe MARL framework for mixed-autonomy platoons. Specifically, this framework (i) characterizes cooperative safety by designing a cooperative Control Barrier Function (CBF), enabling CAVs to collaboratively improve the safety of the entire platoon, (ii) provides a safety guarantee to the MARL-based controller by integrating the CBF-based safety constraints into MARL through a differentiable quadratic programming (QP) layer, and (iii) incorporates a conformal prediction module that enables each CAV to estimate the unknown behaviors of the surrounding vehicles with uncertainty qualification. Simulation results show that our proposed control strategy can effectively enhance the system-level safety through CAV cooperation of a mixed-autonomy platoon with a minimal impact on control performance.