Learning High-Resolution Domain-Specific Representations with a GAN Generator

TL;DR

This paper explores how GAN generator representations can be projected onto semantic maps and used for unsupervised domain-specific pretraining, improving semi-supervised semantic segmentation performance.

Contribution

It introduces the LayerMatch scheme for approximating GAN representations, enabling effective semi-supervised pretraining with minimal labeled data.

Findings

LayerMatch enables semantic projection from GAN representations.

Pretraining with LayerMatch improves semi-supervised segmentation accuracy.

Outperforms standard supervised and recent semi-supervised methods.

Abstract

In recent years generative models of visual data have made a great progress, and now they are able to produce images of high quality and diversity. In this work we study representations learnt by a GAN generator. First, we show that these representations can be easily projected onto semantic segmentation map using a lightweight decoder. We find that such semantic projection can be learnt from just a few annotated images. Based on this finding, we propose LayerMatch scheme for approximating the representation of a GAN generator that can be used for unsupervised domain-specific pretraining. We consider the semi-supervised learning scenario when a small amount of labeled data is available along with a large unlabeled dataset from the same domain. We find that the use of LayerMatch-pretrained backbone leads to superior accuracy compared to standard supervised pretraining on ImageNet. Moreover, this simple approach also outperforms recent semi-supervised semantic segmentation methods that use both labeled and unlabeled data during training. Source code for reproducing our experiments will be available at the time of publication.