InceptionHuman: Controllable Prompt-to-NeRF for Photorealistic 3D Human Generation

TL;DR

InceptionHuman is a novel prompt-to-NeRF framework that enables controllable, photorealistic 3D human generation from multimodal prompts, addressing previous limitations in realism, pose, and view diversity.

Contribution

The paper introduces two innovative modules, IPAR and PAR, that enhance 3D human synthesis by refining pose-aware images and augmenting views using diffusion priors.

Findings

Achieves state-of-the-art quality in 3D human generation

Produces consistent, photorealistic views from multiple angles

Outperforms existing methods in qualitative and quantitative metrics

Abstract

This paper presents InceptionHuman, a prompt-to-NeRF framework that allows easy control via a combination of prompts in different modalities (e.g., text, poses, edge, segmentation map, etc) as inputs to generate photorealistic 3D humans. While many works have focused on generating 3D human models, they suffer one or more of the following: lack of distinctive features, unnatural shading/shadows, unnatural poses/clothes, limited views, etc. InceptionHuman achieves consistent 3D human generation within a progressively refined NeRF space with two novel modules, Iterative Pose-Aware Refinement (IPAR) and Progressive-Augmented Reconstruction (PAR). IPAR iteratively refines the diffusion-generated images and synthesizes high-quality 3D-aware views considering the close-pose RGB values. PAR employs a pretrained diffusion prior to augment the generated synthetic views and adds regularization for view-independent appearance. Overall, the synthesis of photorealistic novel views empowers the resulting 3D human NeRF from 360-degree perspectives. Extensive qualitative and quantitative experimental comparison show that our InceptionHuman models achieve state-of-the-art application quality.