UniC-Lift: Unified 3D Instance Segmentation via Contrastive Learning

TL;DR

UniC-Lift introduces a unified contrastive learning framework for 3D instance segmentation that improves accuracy and efficiency by integrating feature embedding and label decoding, addressing boundary artifacts with a novel hard-mining approach.

Contribution

The paper presents a novel unified framework combining contrastive learning and label decoding for 3D segmentation, reducing training complexity and enhancing performance.

Findings

Outperforms baselines on ScanNet, Replica3D, Messy-Rooms datasets.

Reduces training time compared to two-stage methods.

Addresses boundary artifacts with a new hard-mining technique.

Abstract

3D Gaussian Splatting (3DGS) and Neural Radiance Fields (NeRF) have advanced novel-view synthesis. Recent methods extend multi-view 2D segmentation to 3D, enabling instance/semantic segmentation for better scene understanding. A key challenge is the inconsistency of 2D instance labels across views, leading to poor 3D predictions. Existing methods use a two-stage approach in which some rely on contrastive learning with hyperparameter-sensitive clustering, while others preprocess labels for consistency. We propose a unified framework that merges these steps, reducing training time and improving performance by introducing a learnable feature embedding for segmentation in Gaussian primitives. This embedding is then efficiently decoded into instance labels through a novel "Embedding-to-Label" process, effectively integrating the optimization. While this unified framework offers substantial benefits, we observed artifacts at the object boundaries. To address the object boundary issues, we propose hard-mining samples along these boundaries. However, directly applying hard mining to the feature embeddings proved unstable. Therefore, we apply a linear layer to the rasterized feature embeddings before calculating the triplet loss, which stabilizes training and significantly improves performance. Our method outperforms baselines qualitatively and quantitatively on the ScanNet, Replica3D, and Messy-Rooms datasets.