3D-Aware Semantic-Guided Generative Model for Human Synthesis

TL;DR

This paper introduces 3D-SGAN, a novel model that combines GNeRF and texture generation to produce high-quality, controllable 3D human images from 2D data without extra 3D info.

Contribution

It presents a new 3D-aware generative model for human synthesis that effectively learns 3D representations and realistic textures without additional 3D supervision.

Findings

Outperforms recent baselines on DeepFashion dataset

Produces photo-realistic, controllable 3D human images

Learns 3D human representations without extra 3D data

Abstract

Generative Neural Radiance Field (GNeRF) models, which extract implicit 3D representations from 2D images, have recently been shown to produce realistic images representing rigid/semi-rigid objects, such as human faces or cars. However, they usually struggle to generate high-quality images representing non-rigid objects, such as the human body, which is of a great interest for many computer graphics applications. This paper proposes a 3D-aware Semantic-Guided Generative Model (3D-SGAN) for human image synthesis, which combines a GNeRF with a texture generator. The former learns an implicit 3D representation of the human body and outputs a set of 2D semantic segmentation masks. The latter transforms these semantic masks into a real image, adding a realistic texture to the human appearance. Without requiring additional 3D information, our model can learn 3D human representations with a photo-realistic, controllable generation. Our experiments on the DeepFashion dataset show that 3D-SGAN significantly outperforms the most recent baselines. The code is available at https://github.com/zhangqianhui/3DSGAN