Box2Seg: Learning Semantics of 3D Point Clouds with Box-Level Supervision

TL;DR

This paper introduces Box2Seg, a neural network that learns dense 3D point cloud semantics using only bounding box annotations by generating pseudo labels through geometric and topological analysis.

Contribution

We propose a novel architecture, Box2Seg, that leverages bounding box supervision and pseudo-labeling techniques for efficient 3D semantic segmentation.

Findings

Achieves competitive results on S3DIS and ScanNet benchmarks.

Effectively utilizes cheap bounding box annotations for training.

Outperforms some existing weakly supervised methods.

Abstract

Learning dense point-wise semantics from unstructured 3D point clouds with fewer labels, although a realistic problem, has been under-explored in literature. While existing weakly supervised methods can effectively learn semantics with only a small fraction of point-level annotations, we find that the vanilla bounding box-level annotation is also informative for semantic segmentation of large-scale 3D point clouds. In this paper, we introduce a neural architecture, termed Box2Seg, to learn point-level semantics of 3D point clouds with bounding box-level supervision. The key to our approach is to generate accurate pseudo labels by exploring the geometric and topological structure inside and outside each bounding box. Specifically, an attention-based self-training (AST) technique and Point Class Activation Mapping (PCAM) are utilized to estimate pseudo-labels. The network is further trained and refined with pseudo labels. Experiments on two large-scale benchmarks including S3DIS and ScanNet demonstrate the competitive performance of the proposed method. In particular, the proposed network can be trained with cheap, or even off-the-shelf bounding box-level annotations and subcloud-level tags.