Leveraging GAN Priors for Few-Shot Part Segmentation

TL;DR

This paper introduces a novel method that leverages GAN priors through a pre-training and fine-tuning approach, combined with prompt design and a two-stream architecture, to improve few-shot part segmentation performance.

Contribution

It proposes a new fine-tuning strategy to convert image generators into segmentation generators and introduces a two-stream architecture for better task-specific feature learning.

Findings

Achieves state-of-the-art results on multiple datasets.

Effectively leverages large-scale support images.

Demonstrates the benefit of prompt design in bridging generation and perception tasks.

Abstract

Few-shot part segmentation aims to separate different parts of an object given only a few annotated samples. Due to the challenge of limited data, existing works mainly focus on learning classifiers over pre-trained features, failing to learn task-specific features for part segmentation. In this paper, we propose to learn task-specific features in a "pre-training"-"fine-tuning" paradigm. We conduct prompt designing to reduce the gap between the pre-train task (i.e., image generation) and the downstream task (i.e., part segmentation), so that the GAN priors for generation can be leveraged for segmentation. This is achieved by projecting part segmentation maps into the RGB space and conducting interpolation between RGB segmentation maps and original images. Specifically, we design a fine-tuning strategy to progressively tune an image generator into a segmentation generator, where the supervision of the generator varying from images to segmentation maps by interpolation. Moreover, we propose a two-stream architecture, i.e., a segmentation stream to generate task-specific features, and an image stream to provide spatial constraints. The image stream can be regarded as a self-supervised auto-encoder, and this enables our model to benefit from large-scale support images. Overall, this work is an attempt to explore the internal relevance between generation tasks and perception tasks by prompt designing. Extensive experiments show that our model can achieve state-of-the-art performance on several part segmentation datasets.