REMOT: A Region-to-Whole Framework for Realistic Human Motion Transfer

TL;DR

REMOT introduces a novel GAN-based framework for human motion transfer that generates realistic videos by progressively assembling body parts and aligning global appearance and textures, outperforming existing methods.

Contribution

The paper proposes REMOT, a region-to-whole framework with modules for global and texture alignment, improving realism and reducing artifacts in human motion transfer.

Findings

Achieves state-of-the-art results on public benchmarks.

Effectively reduces artifacts caused by pose and scale differences.

Demonstrates superior visual quality in generated videos.

Abstract

Human Video Motion Transfer (HVMT) aims to, given an image of a source person, generate his/her video that imitates the motion of the driving person. Existing methods for HVMT mainly exploit Generative Adversarial Networks (GANs) to perform the warping operation based on the flow estimated from the source person image and each driving video frame. However, these methods always generate obvious artifacts due to the dramatic differences in poses, scales, and shifts between the source person and the driving person. To overcome these challenges, this paper presents a novel REgionto-whole human MOtion Transfer (REMOT) framework based on GANs. To generate realistic motions, the REMOT adopts a progressive generation paradigm: it first generates each body part in the driving pose without flow-based warping, then composites all parts into a complete person of the driving motion. Moreover, to preserve the natural global appearance, we design a Global Alignment Module to align the scale and position of the source person with those of the driving person based on their layouts. Furthermore, we propose a Texture Alignment Module to keep each part of the person aligned according to the similarity of the texture. Finally, through extensive quantitative and qualitative experiments, our REMOT achieves state-of-the-art results on two public benchmarks.