Spatially Multi-conditional Image Generation

TL;DR

This paper introduces a novel pixel-wise transformer architecture for spatially multi-conditional image generation, effectively handling heterogeneous and sparse labels to improve control and quality of generated images.

Contribution

It proposes a new neural architecture that merges multi-conditional labels into a homogeneous space for better image generation control, addressing heterogeneity and sparsity issues.

Findings

Outperforms state-of-the-art methods on benchmark datasets

Handles label sparsity by dropping missing tokens during training

Demonstrates improved control over generated images

Abstract

In most scenarios, conditional image generation can be thought of as an inversion of the image understanding process. Since generic image understanding involves solving multiple tasks, it is natural to aim at generating images via multi-conditioning. However, multi-conditional image generation is a very challenging problem due to the heterogeneity and the sparsity of the (in practice) available conditioning labels. In this work, we propose a novel neural architecture to address the problem of heterogeneity and sparsity of the spatially multi-conditional labels. Our choice of spatial conditioning, such as by semantics and depth, is driven by the promise it holds for better control of the image generation process. The proposed method uses a transformer-like architecture operating pixel-wise, which receives the available labels as input tokens to merge them in a learned homogeneous space of labels. The merged labels are then used for image generation via conditional generative adversarial training. In this process, the sparsity of the labels is handled by simply dropping the input tokens corresponding to the missing labels at the desired locations, thanks to the proposed pixel-wise operating architecture. Our experiments on three benchmark datasets demonstrate the clear superiority of our method over the state-of-the-art and compared baselines. The source code will be made publicly available.