PoseCrafter: One-Shot Personalized Video Synthesis Following Flexible Pose Control

TL;DR

PoseCrafter is a one-shot personalized video synthesis method that generates high-quality videos following flexible pose controls, leveraging Stable Diffusion and ControlNet, with novel latent editing and temporal attention techniques.

Contribution

It introduces a new one-shot personalized video generation approach with pose control, incorporating latent editing and temporal attention to improve fidelity and identity preservation.

Findings

Outperforms baseline methods on multiple datasets.

Effectively follows diverse pose inputs including different individuals.

Maintains human identity in open-domain videos.

Abstract

In this paper, we introduce PoseCrafter, a one-shot method for personalized video generation following the control of flexible poses. Built upon Stable Diffusion and ControlNet, we carefully design an inference process to produce high-quality videos without the corresponding ground-truth frames. First, we select an appropriate reference frame from the training video and invert it to initialize all latent variables for generation. Then, we insert the corresponding training pose into the target pose sequences to enhance faithfulness through a trained temporal attention module. Furthermore, to alleviate the face and hand degradation resulting from discrepancies between poses of training videos and inference poses, we implement simple latent editing through an affine transformation matrix involving facial and hand landmarks. Extensive experiments on several datasets demonstrate that PoseCrafter achieves superior results to baselines pre-trained on a vast collection of videos under 8 commonly used metrics. Besides, PoseCrafter can follow poses from different individuals or artificial edits and simultaneously retain the human identity in an open-domain training video. Our project page is available at https://ml-gsai.github.io/PoseCrafter-demo/.