Semi-supervised Semantic Segmentation with Prototype-based Consistency Regularization

TL;DR

This paper introduces a semi-supervised semantic segmentation method that uses prototype-based regularization and consistency constraints to improve label propagation across diverse intra-class features, achieving state-of-the-art results.

Contribution

It proposes a novel prototype-based regularization approach with consistency constraints and a prototype maintenance strategy for semi-supervised segmentation.

Findings

Outperforms state-of-the-art methods on Pascal VOC and Cityscapes.

Effectively handles intra-class variation in semi-supervised segmentation.

Demonstrates significant improvements with the proposed regularization and data augmentation techniques.

Abstract

Semi-supervised semantic segmentation requires the model to effectively propagate the label information from limited annotated images to unlabeled ones. A challenge for such a per-pixel prediction task is the large intra-class variation, i.e., regions belonging to the same class may exhibit a very different appearance even in the same picture. This diversity will make the label propagation hard from pixels to pixels. To address this problem, we propose a novel approach to regularize the distribution of within-class features to ease label propagation difficulty. Specifically, our approach encourages the consistency between the prediction from a linear predictor and the output from a prototype-based predictor, which implicitly encourages features from the same pseudo-class to be close to at least one within-class prototype while staying far from the other between-class prototypes. By further incorporating CutMix operations and a carefully-designed prototype maintenance strategy, we create a semi-supervised semantic segmentation algorithm that demonstrates superior performance over the state-of-the-art methods from extensive experimental evaluation on both Pascal VOC and Cityscapes benchmarks.