Semantic Image Synthesis with Spatially-Adaptive Normalization

TL;DR

This paper introduces spatially-adaptive normalization, a novel layer that improves photorealistic image synthesis from semantic layouts by preserving semantic information during normalization, leading to better visual quality and user control.

Contribution

The paper presents a new spatially-adaptive normalization layer that enhances semantic image synthesis by maintaining semantic details during normalization, outperforming previous methods.

Findings

Improved visual fidelity over existing methods.

Better alignment with input semantic layouts.

Enhanced user control over style and semantics.

Abstract

We propose spatially-adaptive normalization, a simple but effective layer for synthesizing photorealistic images given an input semantic layout. Previous methods directly feed the semantic layout as input to the deep network, which is then processed through stacks of convolution, normalization, and nonlinearity layers. We show that this is suboptimal as the normalization layers tend to ``wash away'' semantic information. To address the issue, we propose using the input layout for modulating the activations in normalization layers through a spatially-adaptive, learned transformation. Experiments on several challenging datasets demonstrate the advantage of the proposed method over existing approaches, regarding both visual fidelity and alignment with input layouts. Finally, our model allows user control over both semantic and style. Code is available at https://github.com/NVlabs/SPADE .