MDrive: Benchmarking Closed-Loop Cooperative Driving for End-to-End Multi-agent Systems

TL;DR

MDrive is a comprehensive closed-loop benchmark for cooperative multi-agent driving, highlighting the benefits and challenges of V2X communication in realistic scenarios.

Contribution

It introduces a new benchmark with diverse scenarios and an open-source toolbox, addressing evaluation gaps in multi-agent cooperative driving systems.

Findings

Multi-agent systems outperform single-agent systems in general.

Perception sharing improves perception but not always planning.

Negotiation can help or hinder planning depending on traffic complexity.

Abstract

Vehicle-to-Everything (V2X) communication has emerged as a promising paradigm for autonomous driving, enabling connected agents to share complementary perception information and negotiate with each other to benefit the final planning. Existing V2X benchmarks, however, fall short in two ways: (i) open-loop evaluations fail to capture the inherently closed-loop nature of driving, leading to evaluation gaps, and (ii) current closed-loop evaluations lack behavioral and interactive diversity to reflect real-world driving. Thus, it is still unclear the extent of benefits of multi-agent systems for closed-loop driving. In this paper, we introduce MDrive, a closed-loop cooperative driving benchmark comprising 225 scenarios grounded in both NHTSA pre-crash typologies and real-world V2X datasets. Our benchmark results demonstrate that multi-agent systems are generally better than single-agent counterparts. However, current multi-agent systems still face two important challenges: (i) perception sharing enhances perceptions, but doesn't always translate to better planning; (ii) negotiation improves planning performance but harms it in complex and dense traffic scenarios. MDrive further provides an open-source toolbox for scenario generation, Real2Sim conversion, and human-in-the-loop simulation. Together, MDrive establishes a reproducible foundation for evaluating and improving the generalization and robustness of cooperative driving systems.