Multimodal-Wireless: A Large-Scale Dataset for Sensing and Communication

TL;DR

Multimodal-Wireless is a comprehensive large-scale dataset combining high-resolution communication and sensor data across diverse virtual environments, enabling advanced research in sensing, communication, and perception.

Contribution

The paper introduces a novel, integrated dataset combining multimodal sensing and communication data, built with a customizable pipeline for research in wireless communication and perception.

Findings

Dataset includes 160,000 frames across various scenarios.

High synchronization of communication and sensor data at 100 Hz.

Potential applications demonstrated in beam prediction with multimodal models.

Abstract

This paper presents Multimodal-Wireless, a large-scale open-source dataset for multimodal sensing and communication research. The dataset is generated through an integrated and customizable data pipeline built upon the CARLA simulator and Sionna framework, and features high-resolution communication channel state information (CSI) fully synchronized with five other sensor modalities, namely LiDAR, RGB and depth camera, inertial measurement unit (IMU) and radar, all sampled at 100 Hz. It contains approximately 160,000 frames collected across four virtual towns, sixteen communication scenarios, and three weather conditions. This paper provides a comprehensive overview of the dataset, outlining its key features, overall framework, and technical implementation details. In addition, it explores potential research applications concerning communication and collaborative perception, exemplified by beam prediction using a multimodal large language model. The dataset is open in https://le-liang.github.io/mmw/.