Template-Free Single-View 3D Human Digitalization with Diffusion-Guided LRM

TL;DR

This paper introduces Human-LRM, a diffusion-guided model that reconstructs detailed 3D human models from a single image without templates, improving occlusion handling and realism over previous methods.

Contribution

It combines a single-view LRM model with a diffusion model to enhance occluded details and achieve template-free 3D human reconstruction from a single image.

Findings

Outperforms existing methods in geometry and appearance quality

Effectively enhances occluded parts with realistic details

Achieves superior generalization across diverse datasets

Abstract

Reconstructing 3D humans from a single image has been extensively investigated. However, existing approaches often fall short on capturing fine geometry and appearance details, hallucinating occluded parts with plausible details, and achieving generalization across unseen and in-the-wild datasets. We present Human-LRM, a diffusion-guided feed-forward model that predicts the implicit field of a human from a single image. Leveraging the power of the state-of-the-art reconstruction model (i.e., LRM) and generative model (i.e Stable Diffusion), our method is able to capture human without any template prior, e.g., SMPL, and effectively enhance occluded parts with rich and realistic details. Our approach first uses a single-view LRM model with an enhanced geometry decoder to get the triplane NeRF representation. The novel view renderings from the triplane NeRF provide strong geometry and color prior, from which we generate photo-realistic details for the occluded parts using a diffusion model. The generated multiple views then enable reconstruction with high-quality geometry and appearance, leading to superior overall performance comparing to all existing human reconstruction methods.