DynamicAvatars: Accurate Dynamic Facial Avatars Reconstruction and Precise Editing with Diffusion Models

TL;DR

DynamicAvatars is a novel framework that reconstructs and edits photorealistic dynamic 3D facial avatars from videos, integrating large language models for precise, flexible editing and addressing previous limitations in facial accuracy and control.

Contribution

We introduce a dual-tracking framework with a prompt-based editing model and a control module leveraging LLMs, enabling accurate dynamic avatar reconstruction and fine-grained editing capabilities.

Findings

Achieves photorealistic dynamic 3D avatar reconstruction from videos.

Enables precise, prompt-based editing with improved stability.

Addresses facial distortion and movement inaccuracies in existing methods.

Abstract

Generating and editing dynamic 3D head avatars are crucial tasks in virtual reality and film production. However, existing methods often suffer from facial distortions, inaccurate head movements, and limited fine-grained editing capabilities. To address these challenges, we present DynamicAvatars, a dynamic model that generates photorealistic, moving 3D head avatars from video clips and parameters associated with facial positions and expressions. Our approach enables precise editing through a novel prompt-based editing model, which integrates user-provided prompts with guiding parameters derived from large language models (LLMs). To achieve this, we propose a dual-tracking framework based on Gaussian Splatting and introduce a prompt preprocessing module to enhance editing stability. By incorporating a specialized GAN algorithm and connecting it to our control module, which generates precise guiding parameters from LLMs, we successfully address the limitations of existing methods. Additionally, we develop a dynamic editing strategy that selectively utilizes specific training datasets to improve the efficiency and adaptability of the model for dynamic editing tasks.