VLMFusionOcc3D: VLM Assisted Multi-Modal 3D Semantic Occupancy Prediction

TL;DR

VLMFusionOcc3D introduces a multimodal framework that leverages vision-language models, weather-aware fusion, and geometric alignment to improve 3D semantic occupancy prediction in autonomous driving, especially under adverse weather conditions.

Contribution

The paper presents novel modules like InstVLM, WeathFusion, and DAGA that enhance voxel-based 3D occupancy models by integrating semantic priors, environmental awareness, and geometric consistency.

Findings

Significant performance gains on nuScenes and SemanticKITTI datasets.

Robustness improvements in adverse weather scenarios.

Enhanced semantic accuracy in 3D occupancy prediction.

Abstract

This paper introduces VLMFusionOcc3D, a robust multimodal framework for dense 3D semantic occupancy prediction in autonomous driving. Current voxel-based occupancy models often struggle with semantic ambiguity in sparse geometric grids and performance degradation under adverse weather conditions. To address these challenges, we leverage the rich linguistic priors of Vision-Language Models (VLMs) to anchor ambiguous voxel features to stable semantic concepts. Our framework initiates with a dual-branch feature extraction pipeline that projects multi-view images and LiDAR point clouds into a unified voxel space. We propose Instance-driven VLM Attention (InstVLM), which utilizes gated cross-attention and LoRA-adapted CLIP embeddings to inject high-level semantic and geographic priors directly into the 3D voxels. Furthermore, we introduce Weather-Aware Adaptive Fusion (WeathFusion), a dynamic gating mechanism that utilizes vehicle metadata and weather-conditioned prompts to re-weight sensor contributions based on real-time environmental reliability. To ensure structural consistency, a Depth-Aware Geometric Alignment (DAGA) loss is employed to align dense camera-derived geometry with sparse, spatially accurate LiDAR returns. Extensive experiments on the nuScenes and SemanticKITTI datasets demonstrate that our plug-and-play modules consistently enhance the performance of state-of-the-art voxel-based baselines. Notably, our approach achieves significant improvements in challenging weather scenarios, offering a scalable and robust solution for complex urban navigation.