AniPortraitGAN: Animatable 3D Portrait Generation from 2D Image Collections

TL;DR

AniPortraitGAN introduces a novel 3D-aware GAN capable of generating high-quality, controllable 3D portrait images from 2D collections, enabling facial expression, head pose, and shoulder movement control without 3D or video data.

Contribution

It presents a new generative model that uses a radiance manifold representation with learnable deformations and a dual-camera scheme for high-quality, controllable 3D portraits from 2D images.

Findings

Generates diverse, high-quality 3D portraits

Achieves controllable facial expressions and head poses

Operates without 3D or video data

Abstract

Previous animatable 3D-aware GANs for human generation have primarily focused on either the human head or full body. However, head-only videos are relatively uncommon in real life, and full body generation typically does not deal with facial expression control and still has challenges in generating high-quality results. Towards applicable video avatars, we present an animatable 3D-aware GAN that generates portrait images with controllable facial expression, head pose, and shoulder movements. It is a generative model trained on unstructured 2D image collections without using 3D or video data. For the new task, we base our method on the generative radiance manifold representation and equip it with learnable facial and head-shoulder deformations. A dual-camera rendering and adversarial learning scheme is proposed to improve the quality of the generated faces, which is critical for portrait images. A pose deformation processing network is developed to generate plausible deformations for challenging regions such as long hair. Experiments show that our method, trained on unstructured 2D images, can generate diverse and high-quality 3D portraits with desired control over different properties.