Robust 3D Scene Segmentation through Hierarchical and Learnable Part-Fusion

TL;DR

This paper introduces Segment-Fusion, an attention-based hierarchical fusion method that improves 3D scene segmentation accuracy by addressing part misclassification issues through learnable, flexible, and efficient segment fusion.

Contribution

It proposes a novel learnable attention-based hierarchical fusion approach for 3D segmentation that enhances existing models' performance and reduces part misclassification.

Findings

Achieves up to 5% improvement on ScanNet and S3DIS datasets.

Flexible integration with various segmentation architectures.

Effective combination of graph segmentation, attention fusion, and connected component labeling.

Abstract

3D semantic segmentation is a fundamental building block for several scene understanding applications such as autonomous driving, robotics and AR/VR. Several state-of-the-art semantic segmentation models suffer from the part misclassification problem, wherein parts of the same object are labelled incorrectly. Previous methods have utilized hierarchical, iterative methods to fuse semantic and instance information, but they lack learnability in context fusion, and are computationally complex and heuristic driven. This paper presents Segment-Fusion, a novel attention-based method for hierarchical fusion of semantic and instance information to address the part misclassifications. The presented method includes a graph segmentation algorithm for grouping points into segments that pools point-wise features into segment-wise features, a learnable attention-based network to fuse these segments based on their semantic and instance features, and followed by a simple yet effective connected component labelling algorithm to convert segment features to instance labels. Segment-Fusion can be flexibly employed with any network architecture for semantic/instance segmentation. It improves the qualitative and quantitative performance of several semantic segmentation backbones by upto 5% when evaluated on the ScanNet and S3DIS datasets.