AgileGAN3D: Few-Shot 3D Portrait Stylization by Augmented Transfer Learning

TL;DR

AgileGAN3D introduces a few-shot learning framework that creates personalized 3D stylized portraits from a single photo by leveraging augmented 2D exemplars and guided transfer learning with 3D-aware GANs.

Contribution

The paper presents a novel method combining style prior creation and guided transfer learning to achieve high-quality 3D portrait stylization with minimal data.

Findings

Produces diverse 3D artistic portraits from few exemplars

Maintains subject identity through multi-view consistency loss

Outperforms existing methods in qualitative and quantitative evaluations

Abstract

While substantial progresses have been made in automated 2D portrait stylization, admirable 3D portrait stylization from a single user photo remains to be an unresolved challenge. One primary obstacle here is the lack of high quality stylized 3D training data. In this paper, we propose a novel framework \emph{AgileGAN3D} that can produce 3D artistically appealing and personalized portraits with detailed geometry. New stylization can be obtained with just a few (around 20) unpaired 2D exemplars. We achieve this by first leveraging existing 2D stylization capabilities, \emph{style prior creation}, to produce a large amount of augmented 2D style exemplars. These augmented exemplars are generated with accurate camera pose labels, as well as paired real face images, which prove to be critical for the downstream 3D stylization task. Capitalizing on the recent advancement of 3D-aware GAN models, we perform \emph{guided transfer learning} on a pretrained 3D GAN generator to produce multi-view-consistent stylized renderings. In order to achieve 3D GAN inversion that can preserve subject's identity well, we incorporate \emph{multi-view consistency loss} in the training of our encoder. Our pipeline demonstrates strong capability in turning user photos into a diverse range of 3D artistic portraits. Both qualitative results and quantitative evaluations have been conducted to show the superior performance of our method. Code and pretrained models will be released for reproduction purpose.