SynthSoM-Twin: A Multi-Modal Sensing-Communication Digital-Twin Dataset for Sim2Real Transfer via Synesthesia of Machines
Junlong Chen, Ziwei Huang, Xuesong Cai, Xiang Cheng, and Liuqing Yang

TL;DR
SynthSoM-Twin is a comprehensive multi-modal dataset designed for effective Sim2Real transfer in autonomous systems, validated through cross-modal tasks showing high efficiency with minimal real-world data injection.
Contribution
The paper introduces SynthSoM-Twin, a novel multi-modal sensing-communication dataset with a new framework for extending existing datasets and ensuring spatio-temporal consistency for Sim2Real transfer.
Findings
Models trained on SynthSoM-Twin perform well in real-world tasks.
Injecting less than 15% real-world data achieves comparable or better results.
The dataset enables effective cross-modal generative model training for Sim2Real transfer.
Abstract
This paper constructs a novel multi-modal sensing-communication digital-twin dataset, named SynthSoM-Twin, which is spatio-temporally consistent with the real world, for Sim2Real transfer via Synesthesia of Machines (SoM). To construct the SynthSoM-Twin dataset, we propose a new framework that can extend the quantity and missing modality of existing real-world multi-modal sensing-communication dataset. Specifically, we exploit multi-modal sensing-assisted object detection and tracking algorithms to ensure spatio-temporal consistency of static objects and dynamic objects across real world and simulation environments. The constructed scenario is imported into three high-fidelity simulators, i.e., AirSim, WaveFarer, and Sionna RT. The SynthSoM-Twin dataset contains spatio-temporally consistent data with the real world, including 66,868 snapshots of synthetic RGB images, depth maps, light…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsAdvanced Neural Network Applications · Advanced Wireless Communication Technologies · Face recognition and analysis
