Semantic-shape Adaptive Feature Modulation for Semantic Image Synthesis

TL;DR

This paper introduces a shape-aware feature modulation approach for semantic image synthesis, explicitly encoding object shape to generate more detailed and realistic images, outperforming existing methods.

Contribution

The paper proposes a novel Shape-aware Position Descriptor and Semantic-shape Adaptive Feature Modulation block to incorporate shape information into semantic image synthesis.

Findings

Significantly improves object detail generation.

Outperforms state-of-the-art methods in quantitative metrics.

Enhances the realism of synthesized images.

Abstract

Recent years have witnessed substantial progress in semantic image synthesis, it is still challenging in synthesizing photo-realistic images with rich details. Most previous methods focus on exploiting the given semantic map, which just captures an object-level layout for an image. Obviously, a fine-grained part-level semantic layout will benefit object details generation, and it can be roughly inferred from an object's shape. In order to exploit the part-level layouts, we propose a Shape-aware Position Descriptor (SPD) to describe each pixel's positional feature, where object shape is explicitly encoded into the SPD feature. Furthermore, a Semantic-shape Adaptive Feature Modulation (SAFM) block is proposed to combine the given semantic map and our positional features to produce adaptively modulated features. Extensive experiments demonstrate that the proposed SPD and SAFM significantly improve the generation of objects with rich details. Moreover, our method performs favorably against the SOTA methods in terms of quantitative and qualitative evaluation. The source code and model are available at https://github.com/cszy98/SAFM.