Semantically Annotated Multimodal Dataset for RF Interpretation and Prediction

TL;DR

This paper introduces a multimodal dataset combining RF signals with visual and lidar data to enhance RF interpretation and prediction for wireless AI applications.

Contribution

It presents a new high-quality, multimodal dataset with precise co-registration and annotations to facilitate advanced RF modeling and scene understanding.

Findings

Dataset enables RF heatmap prediction from visual data.

Allows inference of scene semantics from RF signals.

Supports development of RF-based perception models.

Abstract

Current limitations in wireless modeling and radio frequency (RF)-based AI are primarily driven by a lack of high-quality, measurement-based datasets that connect RF signals to their physical environments. RF heatmaps, the typical form of such data, are high-dimensional and complex but lack the geometric and semantic context needed for interpretation, constraining the development of supervised machine learning models. To address this bottleneck, we propose a new class of multimodal datasets that combines RF measurements with auxiliary modalities like high-resolution cameras and lidar to bridge the gap between RF signals and their physical causes. The proposed data collection will span diverse indoor and outdoor environments, featuring both static and dynamic scenarios, including human activities ranging from walking to subtle gestures. By achieving precise spatial and temporal co-registration and creating digital replicas for voxel-level annotation, this dataset will enable transformative AI research. Key tasks include the forward problem of predicting RF heatmaps from visual data to revolutionize wireless system design, and the inverse problem of inferring scene semantics from RF signals, creating a new form of RF-based perception.