UPGPT: Universal Diffusion Model for Person Image Generation, Editing and Pose Transfer

TL;DR

UPGPT introduces a unified diffusion model that integrates text, pose, and visual prompts to generate, edit, and transfer person images consistently, enabling advanced pose and view interpolation.

Contribution

It is the first to unify person image generation, editing, and pose transfer using multimodal diffusion with 3D body model parameters.

Findings

Achieves consistent person appearance across tasks

Enables pose and camera view interpolation

Supports mask-less editing

Abstract

Text-to-image models (T2I) such as StableDiffusion have been used to generate high quality images of people. However, due to the random nature of the generation process, the person has a different appearance e.g. pose, face, and clothing, despite using the same text prompt. The appearance inconsistency makes T2I unsuitable for pose transfer. We address this by proposing a multimodal diffusion model that accepts text, pose, and visual prompting. Our model is the first unified method to perform all person image tasks - generation, pose transfer, and mask-less edit. We also pioneer using small dimensional 3D body model parameters directly to demonstrate new capability - simultaneous pose and camera view interpolation while maintaining the person's appearance.