Topology Enhanced MARL for Multi-Vehicle Cooperative Decision-Making of CAVs

TL;DR

This paper introduces TPE-MARL, a topology-enhanced multi-agent reinforcement learning framework that improves cooperative decision-making for connected autonomous vehicles by reducing state complexity and enhancing exploration.

Contribution

The work develops a game topology tensor for traffic state compression and integrates it with QMIX to enhance MARL performance in CAV coordination.

Findings

TPE-MARL outperforms baseline methods in traffic efficiency and safety.

It achieves decision-making comparable to or better than human drivers.

The approach effectively balances exploration and exploitation in complex traffic scenarios.

Abstract

The exploration-exploitation trade-off constitutes one of the fundamental challenges in reinforcement learning (RL), which is exacerbated in multi-agent reinforcement learning (MARL) due to the exponential growth of joint state-action spaces. This paper proposes a topology-enhanced MARL (TPE-MARL) method for optimizing cooperative decision-making of connected and autonomous vehicles (CAVs) in mixed traffic. This work presents two primary contributions: First, we construct a game topology tensor for dynamic traffic flow, effectively compressing high-dimensional traffic state information and decrease the search space for MARL algorithms. Second, building upon the designed game topology tensor and using QMIX as the backbone RL algorithm, we establish a topology-enhanced MARL framework incorporating visit counts and agent mutual information. Extensive simulations across varying traffic densities and CAV penetration rates demonstrate the effectiveness of TPE-MARL. Evaluations encompassing training dynamics, exploration patterns, macroscopic traffic performance metrics, and microscopic vehicle behaviors reveal that TPE-MARL successfully balances exploration and exploitation. Consequently, it exhibits superior performance in terms of traffic efficiency, safety, decision smoothness, and task completion. Furthermore, the algorithm demonstrates decision-making rationality comparable to or exceeding that of human drivers in both mixed-autonomy and fully autonomous traffic scenarios. Code of our work is available at \href{https://github.com/leoPub/tpemarl}{https://github.com/leoPub/tpemarl}.