Spatially Multi-conditional Image Generation
Ritika Chakraborty, Nikola Popovic, Danda Pani Paudel, Thomas Probst,, Luc Van Gool

TL;DR
This paper introduces a novel pixel-wise transformer architecture for spatially multi-conditional image generation, effectively handling heterogeneous and sparse labels to improve control and quality of generated images.
Contribution
It proposes a new neural architecture that merges multi-conditional labels into a homogeneous space for better image generation control, addressing heterogeneity and sparsity issues.
Findings
Outperforms state-of-the-art methods on benchmark datasets
Handles label sparsity by dropping missing tokens during training
Demonstrates improved control over generated images
Abstract
In most scenarios, conditional image generation can be thought of as an inversion of the image understanding process. Since generic image understanding involves solving multiple tasks, it is natural to aim at generating images via multi-conditioning. However, multi-conditional image generation is a very challenging problem due to the heterogeneity and the sparsity of the (in practice) available conditioning labels. In this work, we propose a novel neural architecture to address the problem of heterogeneity and sparsity of the spatially multi-conditional labels. Our choice of spatial conditioning, such as by semantics and depth, is driven by the promise it holds for better control of the image generation process. The proposed method uses a transformer-like architecture operating pixel-wise, which receives the available labels as input tokens to merge them in a learned homogeneous space…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
Spatially Multi-conditional Image Generation· youtube
Taxonomy
TopicsGenerative Adversarial Networks and Image Synthesis · Image Retrieval and Classification Techniques
