Towards Temporal Fusion Beyond the Field of View for Camera-based Semantic Scene Completion

TL;DR

This paper introduces C3DFusion, a novel temporal fusion module that improves camera-based semantic scene completion by effectively reconstructing out-of-view regions using aligned historical and current frame features, leading to state-of-the-art results.

Contribution

The paper presents C3DFusion, a new module for temporal fusion that explicitly aligns and enhances features from past and current frames to better reconstruct out-of-view regions in SSC.

Findings

C3DFusion outperforms existing methods on SemanticKITTI and SSCBench-KITTI-360 datasets.

It generalizes well across different baseline architectures.

Significant accuracy improvements in out-of-frame region reconstruction.

Abstract

Recent camera-based 3D semantic scene completion (SSC) methods have increasingly explored leveraging temporal cues to enrich the features of the current frame. However, while these approaches primarily focus on enhancing in-frame regions, they often struggle to reconstruct critical out-of-frame areas near the sides of the ego-vehicle, although previous frames commonly contain valuable contextual information about these unseen regions. To address this limitation, we propose the Current-Centric Contextual 3D Fusion (C3DFusion) module, which generates hidden region-aware 3D feature geometry by explicitly aligning 3D-lifted point features from both current and historical frames. C3DFusion performs enhanced temporal fusion through two complementary techniques-historical context blurring and current-centric feature densification-which suppress noise from inaccurately warped historical point features by attenuating their scale, and enhance current point features by increasing their volumetric contribution. Simply integrated into standard SSC architectures, C3DFusion demonstrates strong effectiveness, significantly outperforming state-of-the-art methods on the SemanticKITTI and SSCBench-KITTI-360 datasets. Furthermore, it exhibits robust generalization, achieving notable performance gains when applied to other baseline models.