Diverse Semantic Image Synthesis via Probability Distribution Modeling

TL;DR

This paper introduces a novel framework for diverse semantic image synthesis that models class distributions as continuous probabilities, enabling multimodal and instance-level diverse image generation with improved control and quality.

Contribution

It proposes modeling class-level modulation parameters as continuous distributions and introduces instance-adaptive stochastic sampling and prior noise remapping for enhanced diversity and style control.

Findings

Achieves superior diversity compared to state-of-the-art methods

Maintains comparable image quality

Supports exemplar-based style control

Abstract

Semantic image synthesis, translating semantic layouts to photo-realistic images, is a one-to-many mapping problem. Though impressive progress has been recently made, diverse semantic synthesis that can efficiently produce semantic-level multimodal results, still remains a challenge. In this paper, we propose a novel diverse semantic image synthesis framework from the perspective of semantic class distributions, which naturally supports diverse generation at semantic or even instance level. We achieve this by modeling class-level conditional modulation parameters as continuous probability distributions instead of discrete values, and sampling per-instance modulation parameters through instance-adaptive stochastic sampling that is consistent across the network. Moreover, we propose prior noise remapping, through linear perturbation parameters encoded from paired references, to facilitate supervised training and exemplar-based instance style control at test time. Extensive experiments on multiple datasets show that our method can achieve superior diversity and comparable quality compared to state-of-the-art methods. Code will be available at \url{https://github.com/tzt101/INADE.git}