Cross-identity Video Motion Retargeting with Joint Transformation and Synthesis

TL;DR

This paper introduces TS-Net, a dual-branch network for video motion retargeting that combines transformation and synthesis to produce realistic videos preserving subject identity and motion patterns.

Contribution

The paper presents a novel dual-branch architecture integrating transformation and synthesis for improved video motion retargeting.

Findings

Outperforms state-of-the-art models in face and dance video retargeting

Demonstrates robustness to occlusion and better identity preservation

Achieves superior results on multiple datasets

Abstract

In this paper, we propose a novel dual-branch Transformation-Synthesis network (TS-Net), for video motion retargeting. Given one subject video and one driving video, TS-Net can produce a new plausible video with the subject appearance of the subject video and motion pattern of the driving video. TS-Net consists of a warp-based transformation branch and a warp-free synthesis branch. The novel design of dual branches combines the strengths of deformation-grid-based transformation and warp-free generation for better identity preservation and robustness to occlusion in the synthesized videos. A mask-aware similarity module is further introduced to the transformation branch to reduce computational overhead. Experimental results on face and dance datasets show that TS-Net achieves better performance in video motion retargeting than several state-of-the-art models as well as its single-branch variants. Our code is available at https://github.com/nihaomiao/WACV23_TSNet.