Joint Geometry-Appearance Human Reconstruction in a Unified Latent Space via Bridge Diffusion

TL;DR

This paper presents JGA-LBD, a unified framework that models geometry and appearance of 3D humans in a shared latent space using bridge diffusion, enabling high-fidelity reconstruction from a single RGB image.

Contribution

It introduces a novel joint latent space for geometry and appearance, utilizing bridge diffusion and 3D Gaussian representations for unified 3D human reconstruction.

Findings

Outperforms state-of-the-art in geometry fidelity and appearance quality

Handles challenging in-the-wild scenarios effectively

Enables reconstruction from partial observations

Abstract

Achieving consistent and high-fidelity geometry and appearance reconstruction of 3D digital humans from a single RGB image is inherently a challenging task. Existing studies typically resort to decoupled pipelines for geometry estimation and appearance synthesis, often hindering unified reconstruction and causing inconsistencies. This paper introduces \textbf{JGA-LBD}, a novel framework that unifies the modeling of geometry and appearance into a joint latent representation and formulates the generation process as bridge diffusion. Observing that directly integrating heterogeneous input conditions (e.g., depth maps, SMPL models) leads to substantial training difficulties, we unify all conditions into the 3D Gaussian representations, which can be further compressed into a unified latent space through a shared sparse variational autoencoder (VAE). Subsequently, the specialized form of bridge diffusion enables to start with a partial observation of the target latent code and solely focuses on inferring the missing components. Finally, a dedicated decoding module extracts the complete 3D human geometric structure and renders novel views from the inferred latent representation. Experiments demonstrate that JGA-LBD outperforms current state-of-the-art approaches in terms of both geometry fidelity and appearance quality, including challenging in-the-wild scenarios. Our code will be made publicly available at https://github.com/haiantyz/JGA-LBD.