S4-Net: Geometry-Consistent Semi-Supervised Semantic Segmentation

TL;DR

S4-Net introduces a geometry-based semi-supervised learning approach for semantic segmentation that leverages 3D constraints across multiple views, significantly reducing the need for manual annotations.

Contribution

The paper presents a novel method that enforces 3D geometric consistency in semi-supervised semantic segmentation, enabling high accuracy with minimal manual labels.

Findings

Effective semi-supervised segmentation with few labels

Utilizes 3D geometric constraints across views

Achieves high performance with minimal manual annotations

Abstract

We show that it is possible to learn semantic segmentation from very limited amounts of manual annotations, by enforcing geometric 3D constraints between multiple views. More exactly, image locations corresponding to the same physical 3D point should all have the same label. We show that introducing such constraints during learning is very effective, even when no manual label is available for a 3D point, and can be done simply by employing techniques from 'general' semi-supervised learning to the context of semantic segmentation. To demonstrate this idea, we use RGB-D image sequences of rigid scenes, for a 4-class segmentation problem derived from the ScanNet dataset. Starting from RGB-D sequences with a few annotated frames, we show that we can incorporate RGB-D sequences without any manual annotations to improve the performance, which makes our approach very convenient. Furthermore, we demonstrate our approach for semantic segmentation of objects on the LabelFusion dataset, where we show that one manually labeled image in a scene is sufficient for high performance on the whole scene.