Multi-Agent Reinforcement Learning-based Cooperative Autonomous Driving in Smart Intersections

TL;DR

This paper introduces a cooperative autonomous driving system at intersections using multi-agent reinforcement learning, combining offline pre-training and real-time fine-tuning to improve safety and efficiency.

Contribution

It presents a novel RSU-centric framework with a hybrid RL approach, integrating CQL, BC, and MAPPO with self-attention for effective multi-agent decision-making.

Findings

Achieves less than 0.03% failure rate in complex intersection scenarios.

Outperforms traditional control methods like Autoware.

Demonstrates robustness and generalization across different map configurations.

Abstract

Unsignalized intersections pose significant safety and efficiency challenges due to complex traffic flows. This paper proposes a novel roadside unit (RSU)-centric cooperative driving system leveraging global perception and vehicle-to-infrastructure (V2I) communication. The core of the system is an RSU-based decision-making module using a two-stage hybrid reinforcement learning (RL) framework. At first, policies are pre-trained offline using conservative Q-learning (CQL) combined with behavior cloning (BC) on collected dataset. Subsequently, these policies are fine-tuned in the simulation using multi-agent proximal policy optimization (MAPPO), aligned with a self-attention mechanism to effectively solve inter-agent dependencies. RSUs perform real-time inference based on the trained models to realize vehicle control via V2I communications. Extensive experiments in CARLA environment demonstrate high effectiveness of the proposed system, by: \textit{(i)} achieving failure rates below 0.03\% in coordinating three connected and autonomous vehicles (CAVs) through complex intersection scenarios, significantly outperforming the traditional Autoware control method, and \textit{(ii)} exhibiting strong robustness across varying numbers of controlled agents and shows promising generalization capabilities on other maps.