Weather-Robust Scene Semantics with Vision-Aligned 4D Radar

TL;DR

This paper presents a radar-based scene understanding approach that remains robust in adverse weather by aligning radar data with vision-language models, outperforming camera-based methods in fog and snow conditions.

Contribution

It introduces a novel method to align radar encoders with frozen vision-language models, addressing weather-related robustness in scene semantics.

Findings

Radar configurations outperform camera baseline in fog and snow conditions.

Token-norm mismatch identified as a key failure mode, resolved by LayerNorm.

Analysis reveals tradeoffs in encoder complexity, caption format, and pooling strategy.

Abstract

Cameras and LiDAR degrade in rain, fog, and snow, while millimeter-wave radar remains largely unaffected. We align a radar encoder to frozen SigLIP vision embeddings and decode structured scene captions through a frozen vision-language model (VLM) with approximately 7M trainable parameters. On K-RADAR with held-out fog, light snow, and heavy snow sequences, all radar configurations outperform a camera baseline that collapses to over 90% hallucination. We identify a token-norm mismatch as the dominant failure mode when bridging radar to a frozen VLM and show that projector-output LayerNorm resolves it. Analysis of encoder complexity, caption format, and pooling strategy reveals tradeoffs that inform future radar-VLM pipeline design.