Dual-MTGAN: Stochastic and Deterministic Motion Transfer for Image-to-Video Synthesis

TL;DR

Dual-MTGAN introduces a novel approach for image-to-video synthesis that disentangles content and motion, enabling both deterministic transfer of specific motions and stochastic generation of diverse motion patterns from a single image.

Contribution

It proposes a new GAN model that learns disentangled content and motion representations, allowing flexible motion transfer and diverse motion generation without pre-defined motion features.

Findings

Effective in transferring specific motion patterns from videos to images.

Capable of generating diverse plausible motions from a single image.

Outperforms existing methods in qualitative and quantitative evaluations.

Abstract

Generating videos with content and motion variations is a challenging task in computer vision. While the recent development of GAN allows video generation from latent representations, it is not easy to produce videos with particular content of motion patterns of interest. In this paper, we propose Dual Motion Transfer GAN (Dual-MTGAN), which takes image and video data as inputs while learning disentangled content and motion representations. Our Dual-MTGAN is able to perform deterministic motion transfer and stochastic motion generation. Based on a given image, the former preserves the input content and transfers motion patterns observed from another video sequence, and the latter directly produces videos with plausible yet diverse motion patterns based on the input image. The proposed model is trained in an end-to-end manner, without the need to utilize pre-defined motion features like pose or facial landmarks. Our quantitative and qualitative results would confirm the effectiveness and robustness of our model in addressing such conditioned image-to-video tasks.