USIS: Unsupervised Semantic Image Synthesis

TL;DR

This paper introduces USIS, an unsupervised framework for semantic image synthesis that generates photorealistic images from segmentation masks without requiring paired data, using self-supervised segmentation and wavelet-based discrimination.

Contribution

USIS is the first unsupervised approach to semantic image synthesis that effectively closes the gap with supervised methods by leveraging self-supervision and wavelet-based discrimination.

Findings

USIS outperforms existing unpaired methods on three datasets.

The framework produces multimodal photorealistic images.

It maintains high-frequency details in generated images.

Abstract

Semantic Image Synthesis (SIS) is a subclass of image-to-image translation where a photorealistic image is synthesized from a segmentation mask. SIS has mostly been addressed as a supervised problem. However, state-of-the-art methods depend on a huge amount of labeled data and cannot be applied in an unpaired setting. On the other hand, generic unpaired image-to-image translation frameworks underperform in comparison, because they color-code semantic layouts and feed them to traditional convolutional networks, which then learn correspondences in appearance instead of semantic content. In this initial work, we propose a new Unsupervised paradigm for Semantic Image Synthesis (USIS) as a first step towards closing the performance gap between paired and unpaired settings. Notably, the framework deploys a SPADE generator that learns to output images with visually separable semantic classes using a self-supervised segmentation loss. Furthermore, in order to match the color and texture distribution of real images without losing high-frequency information, we propose to use whole image wavelet-based discrimination. We test our methodology on 3 challenging datasets and demonstrate its ability to generate multimodal photorealistic images with an improved quality in the unpaired setting.