ID-to-3D: Expressive ID-guided 3D Heads via Score Distillation Sampling

TL;DR

ID-to-3D is a novel method that generates detailed, identity-preserving 3D human heads from a single image using diffusion priors and neural representations, enabling high-quality, customizable 3D assets.

Contribution

The paper introduces a new approach combining diffusion models and neural representations to produce expressive, identity-guided 3D heads from minimal input, without large 3D datasets.

Findings

Achieves high-fidelity, identity-consistent 3D head reconstructions.

Generalizes well to unseen identities without extensive 3D data.

Produces detailed geometry and textures suitable for applications like gaming.

Abstract

We propose ID-to-3D, a method to generate identity- and text-guided 3D human heads with disentangled expressions, starting from even a single casually captured in-the-wild image of a subject. The foundation of our approach is anchored in compositionality, alongside the use of task-specific 2D diffusion models as priors for optimization. First, we extend a foundational model with a lightweight expression-aware and ID-aware architecture, and create 2D priors for geometry and texture generation, via fine-tuning only 0.2% of its available training parameters. Then, we jointly leverage a neural parametric representation for the expressions of each subject and a multi-stage generation of highly detailed geometry and albedo texture. This combination of strong face identity embeddings and our neural representation enables accurate reconstruction of not only facial features but also accessories and hair and can be meshed to provide render-ready assets for gaming and telepresence. Our results achieve an unprecedented level of identity-consistent and high-quality texture and geometry generation, generalizing to a ``world'' of unseen 3D identities, without relying on large 3D captured datasets of human assets.