Taming Vision Priors for Data Efficient mmWave Channel Modeling

TL;DR

VisRFTwin is a scalable framework that combines vision-derived priors with differentiable ray tracing to efficiently model mmWave channels, significantly reducing measurement needs and improving accuracy in diverse real-world scenarios.

Contribution

It introduces a novel integration of vision-based material priors with differentiable ray tracing for data-efficient mmWave channel modeling.

Findings

Reduces channel measurement requirements by up to 10 times.

Achieves 59% lower median delay spread error compared to deep learning methods.

Effective across diverse real-world environments.

Abstract

Accurately modeling millimeter-wave (mmWave) propagation is essential for real-time AR and autonomous systems. Differentiable ray tracing offers a physics-grounded solution but still facing deployment challenges due to its over-reliance on exhaustive channel measurements or brittle, hand-tuned scene models for material properties. We present VisRFTwin, a scalable and data-efficient digital-twin framework that integrates vision-derived material priors with differentiable ray tracing. Multi-view images from commodity cameras are processed by a frozen Vision-Language Model to extract dense semantic embeddings, which are translated into initial estimates of permittivity and conductivity for scene surfaces. These priors initialize a Sionna-based differentiable ray tracer, which rapidly calibrates material parameters via gradient descent with only a few dozen sparse channel soundings. Once calibrated, the association between vision features and material parameters is retained, enabling fast transfer to new scenarios without repeated calibration. Evaluations across three real-world scenarios, including office interiors, urban canyons, and dynamic public spaces show that VisRFTwin reduces channel measurement needs by up to 10$\times$ while achieving a 59% lower median delay spread error than pure data-driven deep learning methods.