PersonificationNet: Making customized subject act like a person

TL;DR

PersonificationNet enables fine-grained control of customized subjects like cartoons or plush toys to mimic human poses, enhancing personalized image synthesis with structure alignment techniques.

Contribution

The paper introduces PersonificationNet, a novel framework that aligns the structure of human poses with customized subjects for more realistic pose transfer.

Findings

Outperforms state-of-the-art methods in pose transfer tasks.

Effectively mimics specified subject appearance and pose.

Enhances personalized image generation flexibility.

Abstract

Recently customized generation has significant potential, which uses as few as 3-5 user-provided images to train a model to synthesize new images of a specified subject. Though subsequent applications enhance the flexibility and diversity of customized generation, fine-grained control over the given subject acting like the person's pose is still lack of study. In this paper, we propose a PersonificationNet, which can control the specified subject such as a cartoon character or plush toy to act the same pose as a given referenced person's image. It contains a customized branch, a pose condition branch and a structure alignment module. Specifically, first, the customized branch mimics specified subject appearance. Second, the pose condition branch transfers the body structure information from the human to variant instances. Last, the structure alignment module bridges the structure gap between human and specified subject in the inference stage. Experimental results show our proposed PersonificationNet outperforms the state-of-the-art methods.