Instant 3D Human Avatar Generation using Image Diffusion Models

TL;DR

AvatarPopUp is a fast, high-quality 3D human avatar generation method leveraging diffusion models and 3D lifting, enabling diverse, multimodal control with unprecedented speed.

Contribution

The paper introduces a decoupled diffusion-based approach for rapid, high-quality 3D human avatar creation from images and text, with minimal fine-tuning and broad control capabilities.

Findings

Produces 3D avatars in as little as 2 seconds.

Achieves high-quality, diverse avatars respecting multimodal inputs.

Outperforms existing methods in speed by four orders of magnitude.

Abstract

We present AvatarPopUp, a method for fast, high quality 3D human avatar generation from different input modalities, such as images and text prompts and with control over the generated pose and shape. The common theme is the use of diffusion-based image generation networks that are specialized for each particular task, followed by a 3D lifting network. We purposefully decouple the generation from the 3D modeling which allow us to leverage powerful image synthesis priors, trained on billions of text-image pairs. We fine-tune latent diffusion networks with additional image conditioning for image generation and back-view prediction, and to support qualitatively different multiple 3D hypotheses. Our partial fine-tuning approach allows to adapt the networks for each task without inducing catastrophic forgetting. In our experiments, we demonstrate that our method produces accurate, high-quality 3D avatars with diverse appearance that respect the multimodal text, image, and body control signals. Our approach can produce a 3D model in as few as 2 seconds, a four orders of magnitude speedup wrt the vast majority of existing methods, most of which solve only a subset of our tasks, and with fewer controls. AvatarPopUp enables applications that require the controlled 3D generation of human avatars at scale. The project website can be found at https://www.nikoskolot.com/avatarpopup/.