Towards 3D Object-Centric Feature Learning for Semantic Scene Completion

TL;DR

This paper introduces Ocean, an object-centric framework for 3D semantic scene completion that improves accuracy by focusing on individual object instances and leveraging attention mechanisms, outperforming existing methods.

Contribution

The paper proposes a novel object-centric prediction framework with attention modules and diffusion processes, enhancing semantic scene completion accuracy over ego-centric approaches.

Findings

Achieves state-of-the-art mIoU scores of 17.40 on SemanticKITTI

Achieves state-of-the-art mIoU scores of 20.28 on SSCBench-KITTI360

Demonstrates significant performance improvements over existing methods

Abstract

Vision-based 3D Semantic Scene Completion (SSC) has received growing attention due to its potential in autonomous driving. While most existing approaches follow an ego-centric paradigm by aggregating and diffusing features over the entire scene, they often overlook fine-grained object-level details, leading to semantic and geometric ambiguities, especially in complex environments. To address this limitation, we propose Ocean, an object-centric prediction framework that decomposes the scene into individual object instances to enable more accurate semantic occupancy prediction. Specifically, we first employ a lightweight segmentation model, MobileSAM, to extract instance masks from the input image. Then, we introduce a 3D Semantic Group Attention module that leverages linear attention to aggregate object-centric features in 3D space. To handle segmentation errors and missing instances, we further design a Global Similarity-Guided Attention module that leverages segmentation features for global interaction. Finally, we propose an Instance-aware Local Diffusion module that improves instance features through a generative process and subsequently refines the scene representation in the BEV space. Extensive experiments on the SemanticKITTI and SSCBench-KITTI360 benchmarks demonstrate that Ocean achieves state-of-the-art performance, with mIoU scores of 17.40 and 20.28, respectively.