A Closer Look at Self-training for Zero-Label Semantic Segmentation

TL;DR

This paper introduces a self-training framework with a consistency regularizer for generalized zero-label semantic segmentation, effectively leveraging unlabeled pixels to improve segmentation of both seen and unseen classes.

Contribution

It proposes a novel self-training method with a consistency regularizer to filter pseudo-labels, achieving state-of-the-art results in generalized zero-label semantic segmentation.

Findings

Achieves new state-of-the-art on PascalVOC12 and COCO-stuff datasets.

Outperforms existing methods with more complex strategies.

Effectively segments unseen classes using pseudo-labels.

Abstract

Being able to segment unseen classes not observed during training is an important technical challenge in deep learning, because of its potential to reduce the expensive annotation required for semantic segmentation. Prior zero-label semantic segmentation works approach this task by learning visual-semantic embeddings or generative models. However, they are prone to overfitting on the seen classes because there is no training signal for them. In this paper, we study the challenging generalized zero-label semantic segmentation task where the model has to segment both seen and unseen classes at test time. We assume that pixels of unseen classes could be present in the training images but without being annotated. Our idea is to capture the latent information on unseen classes by supervising the model with self-produced pseudo-labels for unlabeled pixels. We propose a consistency regularizer to filter out noisy pseudo-labels by taking the intersections of the pseudo-labels generated from different augmentations of the same image. Our framework generates pseudo-labels and then retrain the model with human-annotated and pseudo-labelled data. This procedure is repeated for several iterations. As a result, our approach achieves the new state-of-the-art on PascalVOC12 and COCO-stuff datasets in the challenging generalized zero-label semantic segmentation setting, surpassing other existing methods addressing this task with more complex strategies.