Primitive Generation and Semantic-related Alignment for Universal Zero-Shot Segmentation

TL;DR

This paper introduces a novel generative and alignment approach for universal zero-shot segmentation, enabling the recognition of unseen categories across panoptic, instance, and semantic segmentation tasks without training samples.

Contribution

It proposes a primitive-enhanced generative model and semantic-visual feature disentanglement to better transfer semantic relationships to visual features in zero-shot segmentation.

Findings

Achieves state-of-the-art results on zero-shot panoptic segmentation

Effective synthesis of unseen category features using primitives

Improved alignment of semantic and visual feature spaces

Abstract

We study universal zero-shot segmentation in this work to achieve panoptic, instance, and semantic segmentation for novel categories without any training samples. Such zero-shot segmentation ability relies on inter-class relationships in semantic space to transfer the visual knowledge learned from seen categories to unseen ones. Thus, it is desired to well bridge semantic-visual spaces and apply the semantic relationships to visual feature learning. We introduce a generative model to synthesize features for unseen categories, which links semantic and visual spaces as well as addresses the issue of lack of unseen training data. Furthermore, to mitigate the domain gap between semantic and visual spaces, firstly, we enhance the vanilla generator with learned primitives, each of which contains fine-grained attributes related to categories, and synthesize unseen features by selectively assembling these primitives. Secondly, we propose to disentangle the visual feature into the semantic-related part and the semantic-unrelated part that contains useful visual classification clues but is less relevant to semantic representation. The inter-class relationships of semantic-related visual features are then required to be aligned with those in semantic space, thereby transferring semantic knowledge to visual feature learning. The proposed approach achieves impressively state-of-the-art performance on zero-shot panoptic segmentation, instance segmentation, and semantic segmentation. Code is available at https://henghuiding.github.io/PADing/.