Automatic Generation of Semantic Parts for Face Image Synthesis

TL;DR

This paper introduces a neural network architecture that automatically manipulates and generates semantic segmentation masks for face images, enabling automatic control over shape and texture in face image synthesis.

Contribution

The proposed model allows class-wise embedding and editing of masks, enabling automatic shape manipulation for face image synthesis, a feature previously limited to manual editing.

Findings

Model faithfully reconstructs and modifies masks.

Enables automatic shape manipulation in face synthesis.

Supports fully automatic control of shape and texture.

Abstract

Semantic image synthesis (SIS) refers to the problem of generating realistic imagery given a semantic segmentation mask that defines the spatial layout of object classes. Most of the approaches in the literature, other than the quality of the generated images, put effort in finding solutions to increase the generation diversity in terms of style i.e. texture. However, they all neglect a different feature, which is the possibility of manipulating the layout provided by the mask. Currently, the only way to do so is manually by means of graphical users interfaces. In this paper, we describe a network architecture to address the problem of automatically manipulating or generating the shape of object classes in semantic segmentation masks, with specific focus on human faces. Our proposed model allows embedding the mask class-wise into a latent space where each class embedding can be independently edited. Then, a bi-directional LSTM block and a convolutional decoder output a new, locally manipulated mask. We report quantitative and qualitative results on the CelebMask-HQ dataset, which show our model can both faithfully reconstruct and modify a segmentation mask at the class level. Also, we show our model can be put before a SIS generator, opening the way to a fully automatic generation control of both shape and texture. Code available at https://github.com/TFonta/Semantic-VAE.