Morphable Diffusion: 3D-Consistent Diffusion for Single-image Avatar Creation

TL;DR

This paper introduces a novel diffusion-based framework that creates fully 3D-consistent, photorealistic, and controllable human avatars from a single image, enabling realistic view, expression, and pose synthesis.

Contribution

It integrates a 3D morphable model into diffusion models, enabling the first single-image, fully 3D-consistent, animatable human avatar generation.

Findings

Outperforms existing avatar creation models in view synthesis

Enables accurate facial expression and pose control

Produces high-quality, photorealistic avatars from a single image

Abstract

Recent advances in generative diffusion models have enabled the previously unfeasible capability of generating 3D assets from a single input image or a text prompt. In this work, we aim to enhance the quality and functionality of these models for the task of creating controllable, photorealistic human avatars. We achieve this by integrating a 3D morphable model into the state-of-the-art multi-view-consistent diffusion approach. We demonstrate that accurate conditioning of a generative pipeline on the articulated 3D model enhances the baseline model performance on the task of novel view synthesis from a single image. More importantly, this integration facilitates a seamless and accurate incorporation of facial expression and body pose control into the generation process. To the best of our knowledge, our proposed framework is the first diffusion model to enable the creation of fully 3D-consistent, animatable, and photorealistic human avatars from a single image of an unseen subject; extensive quantitative and qualitative evaluations demonstrate the advantages of our approach over existing state-of-the-art avatar creation models on both novel view and novel expression synthesis tasks. The code for our project is publicly available.