Unifying conditional and unconditional semantic image synthesis with OCO-GAN

TL;DR

OCO-GAN is a unified generative model that effectively synthesizes images conditioned on semantic maps or unconditionally, leveraging shared networks and discriminators to outperform specialized models across various datasets and data regimes.

Contribution

The paper introduces OCO-GAN, a novel unified framework that jointly models conditional and unconditional image synthesis with shared networks and adversarial training.

Findings

Outperforms existing hybrid models in all data regimes.

Achieves state-of-the-art results on Cityscapes, COCO-Stuff, ADE20K.

Competitive or superior to specialized models.

Abstract

Generative image models have been extensively studied in recent years. In the unconditional setting, they model the marginal distribution from unlabelled images. To allow for more control, image synthesis can be conditioned on semantic segmentation maps that instruct the generator the position of objects in the image. While these two tasks are intimately related, they are generally studied in isolation. We propose OCO-GAN, for Optionally COnditioned GAN, which addresses both tasks in a unified manner, with a shared image synthesis network that can be conditioned either on semantic maps or directly on latents. Trained adversarially in an end-to-end approach with a shared discriminator, we are able to leverage the synergy between both tasks. We experiment with Cityscapes, COCO-Stuff, ADE20K datasets in a limited data, semi-supervised and full data regime and obtain excellent performance, improving over existing hybrid models that can generate both with and without conditioning in all settings. Moreover, our results are competitive or better than state-of-the art specialised unconditional and conditional models.