Label-free Neural Semantic Image Synthesis

TL;DR

This paper introduces neural semantic image synthesis, a label-free method using neural layouts from pre-trained models for fine-grained spatial control in image generation, outperforming traditional label-based methods.

Contribution

It proposes a novel label-free conditioning approach with neural layouts, providing rich semantic and geometric scene descriptions for diffusion models.

Findings

Neural layout conditioning achieves comparable or better semantic alignment than semantic label maps.

It captures superior semantics, instance separation, and object orientation compared to other label-free methods.

Generated images effectively augment data for perception tasks.

Abstract

Recent work has shown great progress in integrating spatial conditioning to control large, pre-trained text-to-image diffusion models. Despite these advances, existing methods describe the spatial image content using hand-crafted conditioning inputs, which are either semantically ambiguous (e.g., edges) or require expensive manual annotations (e.g., semantic segmentation). To address these limitations, we propose a new label-free way of conditioning diffusion models to enable fine-grained spatial control. We introduce the concept of neural semantic image synthesis, which uses neural layouts extracted from pre-trained foundation models as conditioning. Neural layouts are advantageous as they provide rich descriptions of the desired image, containing both semantics and detailed geometry of the scene. We experimentally show that images synthesized via neural semantic image synthesis achieve similar or superior pixel-level alignment of semantic classes compared to those created using expensive semantic label maps. At the same time, they capture better semantics, instance separation, and object orientation than other label-free conditioning options, such as edges or depth. Moreover, we show that images generated by neural layout conditioning can effectively augment real data for training various perception tasks.