Variation-Aware Semantic Image Synthesis

TL;DR

This paper introduces a variation-aware approach to semantic image synthesis that enhances intra-class diversity, resulting in more natural and photorealistic images, by incorporating semantic noise and position codes.

Contribution

It proposes simple methods to improve intra-class variation in semantic image synthesis, addressing a key limitation of current algorithms.

Findings

Enhanced intra-class variation leads to more natural images.

Achieved slightly better FID and mIoU scores.

Compatible with state-of-the-art algorithms.

Abstract

Semantic image synthesis (SIS) aims to produce photorealistic images aligning to given conditional semantic layout and has witnessed a significant improvement in recent years. Although the diversity in image-level has been discussed heavily, class-level mode collapse widely exists in current algorithms. Therefore, we declare a new requirement for SIS to achieve more photorealistic images, variation-aware, which consists of inter- and intra-class variation. The inter-class variation is the diversity between different semantic classes while the intra-class variation stresses the diversity inside one class. Through analysis, we find that current algorithms elusively embrace the inter-class variation but the intra-class variation is still not enough. Further, we introduce two simple methods to achieve variation-aware semantic image synthesis (VASIS) with a higher intra-class variation, semantic noise and position code. We combine our method with several state-of-the-art algorithms and the experimental result shows that our models generate more natural images and achieves slightly better FIDs and/or mIoUs than the counterparts. Our codes and models will be publicly available.