WeatherOcc3D: VLM-Assisted Adverse Weather Aware 3D Semantic Occupancy Prediction

TL;DR

WeatherOcc3D introduces a VLM-assisted framework that adaptively fuses camera and LiDAR data for 3D semantic occupancy prediction under adverse weather conditions, guided by environmental cues.

Contribution

It leverages pre-trained CLIP embeddings and a gating strategy to dynamically re-weight sensor inputs based on weather-related environmental factors.

Findings

Achieves 26.3 mIoU on nuScenes with OccMamba architecture.

Outperforms traditional baselines significantly.

Effectively handles adverse weather variability in sensor fusion.

Abstract

While multi-modal 3D semantic occupancy prediction typically enhances robustness by fusing camera and LiDAR inputs, its effectiveness is fundamentally constrained by environmental variability. Specifically, camera sensors suffer from severe low-light degradation, while LiDAR sensors encounter significant backscatter noise during heavy precipitation. These adverse conditions create a modality trust problem, as static fusion strategies fail to adaptively re-weight inputs when a specific sensor becomes unreliable. To address this, we propose a VLM-assisted framework leveraging the pre-trained CLIP latent space to guide multi-sensor integration via linguistic environmental cues. We utilize a parameter-efficient adapter to align weather-specific text embeddings with sensor features, coupled with a gating strategy that decomposes environmental uncertainty into two factors: visibility and illumination. This enables the model to dynamically modulate the fusion ratio - prioritizing semantic camera features in clear daylight and shifting to geometric LiDAR priors during rainy nights. Evaluations on the nuScenes dataset demonstrate the versatility of our approach, as implementing our proposed framework on the OccMamba and M-CONet architectures achieves mIoU scores of 26.3 and 21.1, respectively, significantly outperforming their traditional baselines.