HumanCrafter: Synergizing Generalizable Human Reconstruction and Semantic 3D Segmentation

TL;DR

HumanCrafter is a unified framework that jointly models appearance and human-part semantics from a single image, improving 3D human reconstruction and segmentation by integrating priors and enabling cross-task synergy.

Contribution

It introduces a novel multi-task approach with integrated priors and an interactive annotation method for high-quality data, advancing 3D human reconstruction and segmentation.

Findings

Outperforms state-of-the-art in 3D human-part segmentation

Achieves superior 3D reconstruction quality from a single image

Demonstrates effective cross-task synergy and semantic consistency

Abstract

Recent advances in generative models have achieved high-fidelity in 3D human reconstruction, yet their utility for specific tasks (e.g., human 3D segmentation) remains constrained. We propose HumanCrafter, a unified framework that enables the joint modeling of appearance and human-part semantics from a single image in a feed-forward manner. Specifically, we integrate human geometric priors in the reconstruction stage and self-supervised semantic priors in the segmentation stage. To address labeled 3D human datasets scarcity, we further develop an interactive annotation procedure for generating high-quality data-label pairs. Our pixel-aligned aggregation enables cross-task synergy, while the multi-task objective simultaneously optimizes texture modeling fidelity and semantic consistency. Extensive experiments demonstrate that HumanCrafter surpasses existing state-of-the-art methods in both 3D human-part segmentation and 3D human reconstruction from a single image.