Recursive Training for Zero-Shot Semantic Segmentation

TL;DR

This paper introduces a recursive training method with a novel ZS-MMD loss to improve zero-shot semantic segmentation, enabling models to better identify unseen classes with state-of-the-art results.

Contribution

It proposes a recursive training scheme using pseudo-feature supervision and ZS-MMD loss to enhance discriminative features for unseen classes in zero-shot segmentation.

Findings

Achieves state-of-the-art results on Pascal-VOC 2012.

Outperforms previous models in zero-shot segmentation accuracy.

Provides a new training paradigm for zero-shot semantic segmentation.

Abstract

General purpose semantic segmentation relies on a backbone CNN network to extract discriminative features that help classify each image pixel into a 'seen' object class (ie., the object classes available during training) or a background class. Zero-shot semantic segmentation is a challenging task that requires a computer vision model to identify image pixels belonging to an object class which it has never seen before. Equipping a general purpose semantic segmentation model to separate image pixels of 'unseen' classes from the background remains an open challenge. Some recent models have approached this problem by fine-tuning the final pixel classification layer of a semantic segmentation model for a Zero-Shot setting, but struggle to learn discriminative features due to the lack of supervision. We propose a recursive training scheme to supervise the retraining of a semantic segmentation model for a zero-shot setting using a pseudo-feature representation. To this end, we propose a Zero-Shot Maximum Mean Discrepancy (ZS-MMD) loss that weighs high confidence outputs of the pixel classification layer as a pseudo-feature representation, and feeds it back to the generator. By closing-the-loop on the generator end, we provide supervision during retraining that in turn helps the model learn a more discriminative feature representation for 'unseen' classes. We show that using our recursive training and ZS-MMD loss, our proposed model achieves state-of-the-art performance on the Pascal-VOC 2012 dataset and Pascal-Context dataset.