Towards Native Generative Model for 3D Head Avatar

TL;DR

This paper proposes a novel method for learning a native 3D head generative model from limited datasets, enabling 360-degree renderable, editable, and high-quality 3D head avatars suitable for VR and gaming.

Contribution

It introduces a new approach to generate 3D head models with high accuracy from limited data, including effective representation utilization, disentanglement of appearance, shape, and motion, and improved generalization.

Findings

The proposed model produces 360° renderable 3D head avatars.

It successfully disentangles appearance, shape, and motion for editable models.

Experiments demonstrate improved quality and generalization over existing methods.

Abstract

Creating 3D head avatars is a significant yet challenging task for many applicated scenarios. Previous studies have set out to learn 3D human head generative models using massive 2D image data. Although these models are highly generalizable for human appearance, their result models are not 360$^\circ$-renderable, and the predicted 3D geometry is unreliable. Therefore, such results cannot be used in VR, game modeling, and other scenarios that require 360$^\circ$-renderable 3D head models. An intuitive idea is that 3D head models with limited amount but high 3D accuracy are more reliable training data for a high-quality 3D generative model. In this vein, we delve into how to learn a native generative model for 360$^\circ$ full head from a limited 3D head dataset. Specifically, three major problems are studied: 1) how to effectively utilize various representations for generating the 360$^\circ$-renderable human head; 2) how to disentangle the appearance, shape, and motion of human faces to generate a 3D head model that can be edited by appearance and driven by motion; 3) and how to extend the generalization capability of the generative model to support downstream tasks. Comprehensive experiments are conducted to verify the effectiveness of the proposed model. We hope the proposed models and artist-designed dataset can inspire future research on learning native generative 3D head models from limited 3D datasets.