My3DGen: A Scalable Personalized 3D Generative Model

TL;DR

My3DGen is a scalable personalized 3D face generative model that efficiently creates individual-specific 3D face priors using minimal training images, enabling high-quality view synthesis and editing with significantly fewer parameters.

Contribution

It introduces a low-rank decomposition approach to personalize 3D face models with only 240K parameters per individual, reducing training complexity and storage compared to full fine-tuning methods.

Findings

Achieves high-quality personalized 3D face synthesis with as few as 50 images.

Reduces trainable parameters by 127 times compared to full fine-tuning.

Preserves identity and enables semantic editing and novel view synthesis.

Abstract

In recent years, generative 3D face models (e.g., EG3D) have been developed to tackle the problem of synthesizing photo-realistic faces. However, these models are often unable to capture facial features unique to each individual, highlighting the importance of personalization. Some prior works have shown promise in personalizing generative face models, but these studies primarily focus on 2D settings. Also, these methods require both fine-tuning and storing a large number of parameters for each user, posing a hindrance to achieving scalable personalization. Another challenge of personalization is the limited number of training images available for each individual, which often leads to overfitting when using full fine-tuning methods. Our proposed approach, My3DGen, generates a personalized 3D prior of an individual using as few as 50 training images. My3DGen allows for novel view synthesis, semantic editing of a given face (e.g. adding a smile), and synthesizing novel appearances, all while preserving the original person's identity. We decouple the 3D facial features into global features and personalized features by freezing the pre-trained EG3D and training additional personalized weights through low-rank decomposition. As a result, My3DGen introduces only $\textbf{240K}$ personalized parameters per individual, leading to a $\textbf{127}\times$ reduction in trainable parameters compared to the $\textbf{30.6M}$ required for fine-tuning the entire parameter space. Despite this significant reduction in storage, our model preserves identity features without compromising the quality of downstream applications.