OneGAN: Simultaneous Unsupervised Learning of Conditional Image Generation, Foreground Segmentation, and Fine-Grained Clustering

TL;DR

This paper introduces OneGAN, an unsupervised framework that jointly learns conditional image generation, foreground segmentation, clustering, and background editing from unlabeled images, achieving state-of-the-art results.

Contribution

It proposes a novel combined GAN and VAE architecture for simultaneous multi-task learning without annotations, advancing unsupervised image understanding.

Findings

Achieves state-of-the-art results in image generation and segmentation.

Effectively clusters images into a two-level hierarchy.

Enables background and foreground manipulation in an unsupervised setting.

Abstract

We present a method for simultaneously learning, in an unsupervised manner, (i) a conditional image generator, (ii) foreground extraction and segmentation, (iii) clustering into a two-level class hierarchy, and (iv) object removal and background completion, all done without any use of annotation. The method combines a Generative Adversarial Network and a Variational Auto-Encoder, with multiple encoders, generators and discriminators, and benefits from solving all tasks at once. The input to the training scheme is a varied collection of unlabeled images from the same domain, as well as a set of background images without a foreground object. In addition, the image generator can mix the background from one image, with a foreground that is conditioned either on that of a second image or on the index of a desired cluster. The method obtains state of the art results in comparison to the literature methods, when compared to the current state of the art in each of the tasks.