Consistent Human Image and Video Generation with Spatially Conditioned Diffusion

TL;DR

This paper introduces a unified diffusion-based approach for consistent human image and video synthesis that maintains appearance and pose fidelity without domain gaps, using a spatially-conditioned inpainting framework and a causal feature interaction design.

Contribution

The authors propose a novel spatially-conditioned inpainting method with a causal feature interaction framework, enabling unified, flexible, and efficient human image and video generation without additional per-instance fine-tuning.

Findings

Achieves consistent appearance and pose in generated images and videos.

Demonstrates strong generalization to unseen identities and poses.

Outperforms existing methods in quality and consistency.

Abstract

Consistent human-centric image and video synthesis aims to generate images or videos with new poses while preserving appearance consistency with a given reference image, which is crucial for low-cost visual content creation. Recent advances based on diffusion models typically rely on separate networks for reference appearance feature extraction and target visual generation, leading to inconsistent domain gaps between references and targets. In this paper, we frame the task as a spatially-conditioned inpainting problem, where the target image is inpainted to maintain appearance consistency with the reference. This approach enables the reference features to guide the generation of pose-compliant targets within a unified denoising network, thereby mitigating domain gaps. Additionally, to better maintain the reference appearance information, we impose a causal feature interaction framework, in which reference features can only query from themselves, while target features can query appearance information from both the reference and the target. To further enhance computational efficiency and flexibility, in practical implementation, we decompose the spatially-conditioned generation process into two stages: reference appearance extraction and conditioned target generation. Both stages share a single denoising network, with interactions restricted to self-attention layers. This proposed method ensures flexible control over the appearance of generated human images and videos. By fine-tuning existing base diffusion models on human video data, our method demonstrates strong generalization to unseen human identities and poses without requiring additional per-instance fine-tuning. Experimental results validate the effectiveness of our approach, showing competitive performance compared to existing methods for consistent human image and video synthesis.