Sim2Radar: Toward Bridging the Radar Sim-to-Real Gap with VLM-Guided Scene Reconstruction

TL;DR

Sim2Radar introduces a novel framework that synthesizes realistic radar data from RGB images using scene reconstruction and physics-based simulation, significantly enhancing radar perception models with limited real data.

Contribution

The paper presents a scalable, end-to-end method for generating synthetic radar data from RGB images, bridging the radar sim-to-real gap without manual scene modeling.

Findings

Synthetic data improves 3D radar detection accuracy by up to +3.7 AP.

Physics-based simulation provides effective geometric priors for radar learning.

Transfer learning from synthetic to real radar data enhances perception performance.

Abstract

Millimeter-wave (mmWave) radar provides reliable perception in visually degraded indoor environments (e.g., smoke, dust, and low light), but learning-based radar perception is bottlenecked by the scarcity and cost of collecting and annotating large-scale radar datasets. We present Sim2Radar, an end-to-end framework that synthesizes training radar data directly from single-view RGB images, enabling scalable data generation without manual scene modeling. Sim2Radar reconstructs a material-aware 3D scene by combining monocular depth estimation, segmentation, and vision-language reasoning to infer object materials, then simulates mmWave propagation with a configurable physics-based ray tracer using Fresnel reflection models parameterized by ITU-R electromagnetic properties. Evaluated on real-world indoor scenes, Sim2Radar improves downstream 3D radar perception via transfer learning: pre-training a radar point-cloud object detection model on synthetic data and fine-tuning on real radar yields up to +3.7 3D AP (IoU 0.3), with gains driven primarily by improved spatial localization. These results suggest that physics-based, vision-driven radar simulation can provide effective geometric priors for radar learning and measurably improve performance under limited real-data supervision.