Multi-V2X: A Large Scale Multi-modal Multi-penetration-rate Dataset for Cooperative Perception

TL;DR

Multi-V2X is a large-scale, multi-modal dataset designed for cooperative perception in V2X systems, addressing limitations of existing datasets by including various penetration rates and multiple agent types to facilitate research in autonomous vehicle perception.

Contribution

The paper introduces Multi-V2X, a comprehensive dataset with multi-modal data and variable penetration rates, enabling advanced research in cooperative perception for autonomous vehicles.

Findings

Provides a large-scale dataset with 549k RGB and 146k LiDAR frames.

Includes scenarios with up to 86.21% CAV penetration rate.

Establishes benchmarks for cooperative 3D object detection.

Abstract

Cooperative perception through vehicle-to-everything (V2X) has garnered significant attention in recent years due to its potential to overcome occlusions and enhance long-distance perception. Great achievements have been made in both datasets and algorithms. However, existing real-world datasets are limited by the presence of few communicable agents, while synthetic datasets typically cover only vehicles. More importantly, the penetration rate of connected and autonomous vehicles (CAVs) , a critical factor for the deployment of cooperative perception technologies, has not been adequately addressed. To tackle these issues, we introduce Multi-V2X, a large-scale, multi-modal, multi-penetration-rate dataset for V2X perception. By co-simulating SUMO and CARLA, we equip a substantial number of cars and roadside units (RSUs) in simulated towns with sensor suites, and collect comprehensive sensing data. Datasets with specified CAV penetration rates can be obtained by masking some equipped cars as normal vehicles. In total, our Multi-V2X dataset comprises 549k RGB frames, 146k LiDAR frames, and 4,219k annotated 3D bounding boxes across six categories. The highest possible CAV penetration rate reaches 86.21%, with up to 31 agents in communication range, posing new challenges in selecting agents to collaborate with. We provide comprehensive benchmarks for cooperative 3D object detection tasks. Our data and code are available at https://github.com/RadetzkyLi/Multi-V2X .