Sketchy Bounding-box Supervision for 3D Instance Segmentation

TL;DR

This paper introduces Sketchy-3DIS, a weakly supervised 3D instance segmentation method that effectively learns from imprecise bounding boxes by jointly training a pseudo labeler and segmentator, achieving state-of-the-art results.

Contribution

The paper proposes a novel framework that improves 3D instance segmentation using sketchy bounding boxes by jointly learning pseudo labels and segmentation, outperforming existing methods.

Findings

Achieves state-of-the-art results on ScanNetV2 and S3DIS benchmarks.

Outperforms several fully supervised methods with sketchy bounding boxes.

Effectively learns from imprecise bounding box annotations.

Abstract

Bounding box supervision has gained considerable attention in weakly supervised 3D instance segmentation. While this approach alleviates the need for extensive point-level annotations, obtaining accurate bounding boxes in practical applications remains challenging. To this end, we explore the inaccurate bounding box, named sketchy bounding box, which is imitated through perturbing ground truth bounding box by adding scaling, translation, and rotation. In this paper, we propose Sketchy-3DIS, a novel weakly 3D instance segmentation framework, which jointly learns pseudo labeler and segmentator to improve the performance under the sketchy bounding-box supervisions. Specifically, we first propose an adaptive box-to-point pseudo labeler that adaptively learns to assign points located in the overlapped parts between two sketchy bounding boxes to the correct instance, resulting in compact and pure pseudo instance labels. Then, we present a coarse-to-fine instance segmentator that first predicts coarse instances from the entire point cloud and then learns fine instances based on the region of coarse instances. Finally, by using the pseudo instance labels to supervise the instance segmentator, we can gradually generate high-quality instances through joint training. Extensive experiments show that our method achieves state-of-the-art performance on both the ScanNetV2 and S3DIS benchmarks, and even outperforms several fully supervised methods using sketchy bounding boxes. Code is available at https://github.com/dengq7/Sketchy-3DIS.