Gaussian Based Adaptive Multi-Modal 3D Semantic Occupancy Prediction

TL;DR

This paper introduces a Gaussian-based adaptive multimodal 3D occupancy prediction model that effectively combines camera and LiDAR data for autonomous vehicles, addressing computational and environmental challenges.

Contribution

It proposes a novel Gaussian-based framework with four components that enhance multimodal fusion, efficiency, and robustness in 3D semantic occupancy prediction.

Findings

Improved accuracy in 3D occupancy prediction

Reduced computational complexity

Enhanced robustness to environmental variations

Abstract

The sparse object detection paradigm shift towards dense 3D semantic occupancy prediction is necessary for dealing with long-tail safety challenges for autonomous vehicles. Nonetheless, the current voxelization methods commonly suffer from excessive computation complexity demands, where the fusion process is brittle, static, and breaks down under dynamic environmental settings. To this end, this research work enhances a novel Gaussian-based adaptive camera-LiDAR multimodal 3D occupancy prediction model that seamlessly bridges the semantic strengths of camera modality with the geometric strengths of LiDAR modality through a memory-efficient 3D Gaussian model. The proposed solution has four key components: (1) LiDAR Depth Feature Aggregation (LDFA), where depth-wise deformable sampling is employed for dealing with geometric sparsity, (2) Entropy-Based Feature Smoothing, where cross-entropy is employed for handling domain-specific noise, (3) Adaptive Camera-LiDAR Fusion, where dynamic recalibration of sensor outputs is performed based on model outputs, and (4) Gauss-Mamba Head that uses Selective State Space Models for global context decoding that enjoys linear computation complexity.