Open-world Point Cloud Semantic Segmentation: A Human-in-the-loop Framework

TL;DR

This paper introduces HOW-Seg, a human-in-the-loop framework for open-world point cloud segmentation that leverages sparse human annotations and hierarchical prototype refinement to effectively segment both known and novel classes.

Contribution

The paper presents the first human-in-the-loop approach for open-world point cloud segmentation, utilizing prototype construction on query data and hierarchical disambiguation to improve segmentation accuracy.

Findings

Achieves state-of-the-art results with sparse annotations.

Outperforms existing methods on S3DIS and ScanNetv2 datasets.

Effectively segments novel classes with minimal human input.

Abstract

Open-world point cloud semantic segmentation (OW-Seg) aims to predict point labels of both base and novel classes in real-world scenarios. However, existing methods rely on resource-intensive offline incremental learning or densely annotated support data, limiting their practicality. To address these limitations, we propose HOW-Seg, the first human-in-the-loop framework for OW-Seg. Specifically, we construct class prototypes, the fundamental segmentation units, directly on the query data, avoiding the prototype bias caused by intra-class distribution shifts between the support and query data. By leveraging sparse human annotations as guidance, HOW-Seg enables prototype-based segmentation for both base and novel classes. Considering the lack of granularity of initial prototypes, we introduce a hierarchical prototype disambiguation mechanism to refine ambiguous prototypes, which correspond to annotations of different classes. To further enrich contextual awareness, we employ a dense conditional random field (CRF) upon the refined prototypes to optimize their label assignments. Through iterative human feedback, HOW-Seg dynamically improves its predictions, achieving high-quality segmentation for both base and novel classes. Experiments demonstrate that with sparse annotations (e.g., one-novel-class-one-click), HOW-Seg matches or surpasses the state-of-the-art generalized few-shot segmentation (GFS-Seg) method under the 5-shot setting. When using advanced backbones (e.g., Stratified Transformer) and denser annotations (e.g., 10 clicks per sub-scene), HOW-Seg achieves 85.27% mIoU on S3DIS and 66.37% mIoU on ScanNetv2, significantly outperforming alternatives.