HuGeDiff: 3D Human Generation via Diffusion with Gaussian Splatting

TL;DR

HuGeDiff introduces a weakly supervised pipeline for 3D human generation that combines image synthesis, feature-to-3D mapping, and diffusion models, achieving faster and more realistic results aligned with text prompts.

Contribution

The paper presents a novel pipeline integrating diffusion models and transformer-based mapping for controllable, realistic 3D human generation from text prompts, with significant speed improvements.

Findings

Orders-of-magnitude faster than previous methods

Improved realism and text-prompt alignment

Enhanced control over human attributes

Abstract

3D human generation is an important problem with a wide range of applications in computer vision and graphics. Despite recent progress in generative AI such as diffusion models or rendering methods like Neural Radiance Fields or Gaussian Splatting, controlling the generation of accurate 3D humans from text prompts remains an open challenge. Current methods struggle with fine detail, accurate rendering of hands and faces, human realism, and controlability over appearance. The lack of diversity, realism, and annotation in human image data also remains a challenge, hindering the development of a foundational 3D human model. We present a weakly supervised pipeline that tries to address these challenges. In the first step, we generate a photorealistic human image dataset with controllable attributes such as appearance, race, gender, etc using a state-of-the-art image diffusion model. Next, we propose an efficient mapping approach from image features to 3D point clouds using a transformer-based architecture. Finally, we close the loop by training a point-cloud diffusion model that is conditioned on the same text prompts used to generate the original samples. We demonstrate orders-of-magnitude speed-ups in 3D human generation compared to the state-of-the-art approaches, along with significantly improved text-prompt alignment, realism, and rendering quality. We will make the code and dataset available.