PoseCraft: Tokenized 3D Body Landmark and Camera Conditioning for Photorealistic Human Image Synthesis

TL;DR

PoseCraft introduces a diffusion-based framework that encodes 3D landmarks and camera parameters as tokens, enabling photorealistic human image synthesis with explicit pose and camera control, surpassing prior methods in quality and detail.

Contribution

The paper proposes a novel tokenized 3D conditioning approach for diffusion models, avoiding 2D re-projection issues and improving photorealism in human image synthesis.

Findings

Achieves higher perceptual quality than existing diffusion methods.

Maintains detailed fabric and hair features better than volumetric SOTA.

Handles large pose and viewpoint variations effectively.

Abstract

Digitizing humans and synthesizing photorealistic avatars with explicit 3D pose and camera controls are central to VR, telepresence, and entertainment. Existing skinning-based workflows require laborious manual rigging or template-based fittings, while neural volumetric methods rely on canonical templates and re-optimization for each unseen pose. We present PoseCraft, a diffusion framework built around tokenized 3D interface: instead of relying only on rasterized geometry as 2D control images, we encode sparse 3D landmarks and camera extrinsics as discrete conditioning tokens and inject them into diffusion via cross-attention. Our approach preserves 3D semantics by avoiding 2D re-projection ambiguity under large pose and viewpoint changes, and produces photorealistic imagery that faithfully captures identity and appearance. To train and evaluate at scale, we also implement GenHumanRF, a data generation workflow that renders diverse supervision from volumetric reconstructions. Our experiments show that PoseCraft achieves significant perceptual quality improvement over diffusion-centric methods, and attains better or comparable metrics to latest volumetric rendering SOTA while better preserving fabric and hair details.