3D Guided Weakly Supervised Semantic Segmentation

TL;DR

This paper introduces a weakly supervised 2D semantic segmentation approach that leverages sparse bounding box labels and 3D information to generate accurate pixel-wise segment proposals, reducing annotation effort.

Contribution

The novel integration of 3D data with bounding box supervision to improve weakly supervised semantic segmentation performance.

Findings

Effective segment proposal generation with limited bounding box labels.

Outperforms recent state-of-the-art methods on 2D-3D-S dataset.

Recursive refinement improves segmentation accuracy.

Abstract

Pixel-wise clean annotation is necessary for fully-supervised semantic segmentation, which is laborious and expensive to obtain. In this paper, we propose a weakly supervised 2D semantic segmentation model by incorporating sparse bounding box labels with available 3D information, which is much easier to obtain with advanced sensors. We manually labeled a subset of the 2D-3D Semantics(2D-3D-S) dataset with bounding boxes, and introduce our 2D-3D inference module to generate accurate pixel-wise segment proposal masks. Guided by 3D information, we first generate a point cloud of objects and calculate objectness probability score for each point. Then we project the point cloud with objectness probabilities back to 2D images followed by a refinement step to obtain segment proposals, which are treated as pseudo labels to train a semantic segmentation network. Our method works in a recursive manner to gradually refine the above-mentioned segment proposals. Extensive experimental results on the 2D-3D-S dataset show that the proposed method can generate accurate segment proposals when bounding box labels are available on only a small subset of training images. Performance comparison with recent state-of-the-art methods further illustrates the effectiveness of our method.