Weighting Pseudo-Labels via High-Activation Feature Index Similarity and Object Detection for Semi-Supervised Segmentation

TL;DR

This paper introduces a novel semi-supervised segmentation method that combines object detection and feature similarity to reliably select and weight pseudo-labels, significantly improving performance on Cityscapes and Pascal VOC datasets.

Contribution

It unifies object detection and segmentation predictions to identify reliable pseudo-labels and uses a noise-robust similarity metric to assign adaptive weights, enhancing semi-supervised segmentation.

Findings

Improves semi-supervised segmentation accuracy on Cityscapes and Pascal VOC.

Effectively filters out noisy pseudo-labels using a combined detection and feature similarity approach.

Enhances existing frameworks with a robust pseudo-label weighting strategy.

Abstract

Semi-supervised semantic segmentation methods leverage unlabeled data by pseudo-labeling them. Thus the success of these methods hinges on the reliablility of the pseudo-labels. Existing methods mostly choose high-confidence pixels in an effort to avoid erroneous pseudo-labels. However, high confidence does not guarantee correct pseudo-labels especially in the initial training iterations. In this paper, we propose a novel approach to reliably learn from pseudo-labels. First, we unify the predictions from a trained object detector and a semantic segmentation model to identify reliable pseudo-label pixels. Second, we assign different learning weights to pseudo-labeled pixels to avoid noisy training signals. To determine these weights, we first use the reliable pseudo-label pixels identified from the first step and labeled pixels to construct a prototype for each class. Then, the per-pixel weight is the structural similarity between the pixel and the prototype measured via rank-statistics similarity. This metric is robust to noise, making it better suited for comparing features from unlabeled images, particularly in the initial training phases where wrong pseudo labels are prone to occur. We show that our method can be easily integrated into four semi-supervised semantic segmentation frameworks, and improves them in both Cityscapes and Pascal VOC datasets.