COIN: Collaborative Interaction-Aware Multi-Agent Reinforcement Learning for Self-Driving Systems

TL;DR

This paper introduces COIN, a novel multi-agent reinforcement learning framework for self-driving cars that enhances collaboration and safety in dense urban traffic through a new algorithm and interaction-aware critic architecture.

Contribution

The paper proposes COIN, a new MARL framework with a novel CIG-TD3 algorithm and dual-level critic for improved collaboration and safety in autonomous driving systems.

Findings

COIN outperforms baseline methods in safety and efficiency.

The dual-level critic improves credit assignment in dense traffic.

Validated through simulations and real-world robot demos.

Abstract

Multi-Agent Self-Driving (MASD) systems provide an effective solution for coordinating autonomous vehicles to reduce congestion and enhance both safety and operational efficiency in future intelligent transportation systems. Multi-Agent Reinforcement Learning (MARL) has emerged as a promising approach for developing advanced end-to-end MASD systems. However, achieving efficient and safe collaboration in dynamic MASD systems remains a significant challenge in dense scenarios with complex agent interactions. To address this challenge, we propose a novel collaborative(CO-) interaction-aware(-IN) MARL framework, named COIN. Specifically, we develop a new counterfactual individual-global twin delayed deep deterministic policy gradient (CIG-TD3) algorithm, crafted in a "centralized training, decentralized execution" (CTDE) manner, which aims to jointly optimize the individual objectives (navigation) and the global objectives (collaboration) of agents. We further introduce a dual-level interaction-aware centralized critic architecture that captures both local pairwise interactions and global system-level dependencies, enabling more accurate global value estimation and improved credit assignment for collaborative policy learning. We conduct extensive simulation experiments in dense urban traffic environments, which demonstrate that COIN consistently outperforms other advanced baseline methods in both safety and efficiency across various system sizes. These results highlight its superiority in complex and dynamic MASD scenarios, as further validated through real-world robot demonstrations. Supplementary videos are available at https://marmotlab.github.io/COIN/