Hierarchical Context Alignment with Disentangled Geometric and Temporal Modeling for Semantic Occupancy Prediction

TL;DR

This paper introduces Hi-SOP, a hierarchical context alignment method that disentangles geometric and temporal features for more accurate semantic occupancy prediction in 3D scenes, improving reliability and performance.

Contribution

The paper proposes a novel hierarchical alignment framework that separately aligns geometric and temporal contexts, enhancing semantic occupancy prediction accuracy and robustness.

Findings

Outperforms state-of-the-art methods on SemanticKITTI and NuScenes datasets.

Improves semantic scene completion and LiDAR segmentation accuracy.

Demonstrates reliable contextual fusion through disentangled alignment.

Abstract

Camera-based 3D Semantic Occupancy Prediction (SOP) is crucial for understanding complex 3D scenes from limited 2D image observations. Existing SOP methods typically aggregate contextual features to assist the occupancy representation learning, alleviating issues like occlusion or ambiguity. However, these solutions often face misalignment issues wherein the corresponding features at the same position across different frames may have different semantic meanings during the aggregation process, which leads to unreliable contextual fusion results and an unstable representation learning process. To address this problem, we introduce a new Hierarchical context alignment paradigm for a more accurate SOP (Hi-SOP). Hi-SOP first disentangles the geometric and temporal context for separate alignment, which two branches are then composed to enhance the reliability of SOP. This parsing of the visual input into a local-global alignment hierarchy includes: (I) disentangled geometric and temporal separate alignment, within each leverages depth confidence and camera pose as prior for relevant feature matching respectively; (II) global alignment and composition of the transformed geometric and temporal volumes based on semantics consistency. Our method outperforms SOTAs for semantic scene completion on the SemanticKITTI & NuScenes-Occupancy datasets and LiDAR semantic segmentation on the NuScenes dataset. The project website is available at https://arlo0o.github.io/hisop.github.io/.