HybridOcc: NeRF Enhanced Transformer-based Multi-Camera 3D Occupancy Prediction

TL;DR

HybridOcc introduces a novel hybrid Transformer and NeRF-based approach for 3D scene completion, effectively inferring both visible and occluded scene geometry in autonomous driving scenarios.

Contribution

The paper proposes HybridOcc, combining Transformer and NeRF representations within a coarse-to-fine framework for improved 3D occupancy prediction.

Findings

Outperforms existing SSC methods on nuScenes and SemanticKITTI datasets.

Effectively infers occluded scene geometry including invisible voxels.

Introduces occupancy-aware ray sampling for enhanced scene understanding.

Abstract

Vision-based 3D semantic scene completion (SSC) describes autonomous driving scenes through 3D volume representations. However, the occlusion of invisible voxels by scene surfaces poses challenges to current SSC methods in hallucinating refined 3D geometry. This paper proposes HybridOcc, a hybrid 3D volume query proposal method generated by Transformer framework and NeRF representation and refined in a coarse-to-fine SSC prediction framework. HybridOcc aggregates contextual features through the Transformer paradigm based on hybrid query proposals while combining it with NeRF representation to obtain depth supervision. The Transformer branch contains multiple scales and uses spatial cross-attention for 2D to 3D transformation. The newly designed NeRF branch implicitly infers scene occupancy through volume rendering, including visible and invisible voxels, and explicitly captures scene depth rather than generating RGB color. Furthermore, we present an innovative occupancy-aware ray sampling method to orient the SSC task instead of focusing on the scene surface, further improving the overall performance. Extensive experiments on nuScenes and SemanticKITTI datasets demonstrate the effectiveness of our HybridOcc on the SSC task.