InstanceBEV: Unifying Instance and BEV Representation for 3D Panoptic Segmentation

TL;DR

InstanceBEV unifies instance and BEV representations to improve 3D panoptic segmentation efficiency and accuracy in autonomous driving, enabling effective multi-task learning with only 8 frames.

Contribution

The paper introduces InstanceBEV, a novel approach that combines map-centric and object-centric methods to enhance 3D perception in BEV space, addressing efficiency and integration challenges.

Findings

Achieves 15.3 RayPQ and 38.2 RayIoU on OCC3D-nuScenes with 8 frames.

Outperforms SparseOcc by 9.3% in RayPQ and 10.7% in RayIoU.

Enables multi-task learning without additional modules.

Abstract

BEV-based 3D perception has emerged as a focal point of research in end-to-end autonomous driving. However, existing BEV approaches encounter significant challenges due to the large feature space, complicating efficient modeling and hindering effective integration of global attention mechanisms. We propose a novel modeling strategy, called InstanceBEV, that synergistically combines the strengths of both map-centric approaches and object-centric approaches. Our method effectively extracts instance-level features within the BEV features, facilitating the implementation of global attention modeling in a highly compressed feature space, thereby addressing the efficiency challenges inherent in map-centric global modeling. Furthermore, our approach enables effective multi-task learning without introducing additional module. We validate the efficiency and accuracy of the proposed model through predicting occupancy, achieving 3D occupancy panoptic segmentation by combining instance information. Experimental results on the OCC3D-nuScenes dataset demonstrate that InstanceBEV, utilizing only 8 frames, achieves a RayPQ of 15.3 and a RayIoU of 38.2. This surpasses SparseOcc's RayPQ by 9.3% and RayIoU by 10.7%, showcasing the effectiveness of multi-task synergy.