Weakly Supervised Semantic Segmentation in 3D Graph-Structured Point Clouds of Wild Scenes

TL;DR

This paper introduces a novel deep graph convolutional network framework that leverages only 2D supervision to perform large-scale 3D semantic segmentation of point clouds in wild scenes, reducing the need for extensive 3D labels.

Contribution

The proposed framework uniquely uses 2D supervision for 3D segmentation, incorporating a Graph-based Pyramid Feature Network and an Observability Network to handle occlusion and structure inference.

Findings

Achieves comparable performance to fully supervised methods on SUNCG and S3DIS datasets.

Effectively handles occlusion and complex spatial relations in 3D scenes.

Reduces reliance on costly 3D annotations.

Abstract

The deficiency of 3D segmentation labels is one of the main obstacles to effective point cloud segmentation, especially for scenes in the wild with varieties of different objects. To alleviate this issue, we propose a novel deep graph convolutional network-based framework for large-scale semantic scene segmentation in point clouds with sole 2D supervision. Different with numerous preceding multi-view supervised approaches focusing on single object point clouds, we argue that 2D supervision is capable of providing sufficient guidance information for training 3D semantic segmentation models of natural scene point clouds while not explicitly capturing their inherent structures, even with only single view per training sample. Specifically, a Graph-based Pyramid Feature Network (GPFN) is designed to implicitly infer both global and local features of point sets and an Observability Network (OBSNet) is introduced to further solve object occlusion problem caused by complicated spatial relations of objects in 3D scenes. During the projection process, perspective rendering and semantic fusion modules are proposed to provide refined 2D supervision signals for training along with a 2D-3D joint optimization strategy. Extensive experimental results demonstrate the effectiveness of our 2D supervised framework, which achieves comparable results with the state-of-the-art approaches trained with full 3D labels, for semantic point cloud segmentation on the popular SUNCG synthetic dataset and S3DIS real-world dataset.