C2F-FWN: Coarse-to-Fine Flow Warping Network for Spatial-Temporal Consistent Motion Transfer

TL;DR

This paper introduces C2F-FWN, a novel neural network architecture that enhances human video motion transfer by improving spatial and temporal consistency through coarse-to-fine flow warping, deformable convolution, and temporal loss functions.

Contribution

The paper proposes a new HVMT method combining flow warping, deformable convolution, and temporal loss to achieve better spatial-temporal consistency and supports flexible appearance editing.

Findings

Outperforms existing methods in spatial and temporal consistency.

Achieves high-quality motion transfer on public and new SoloDance datasets.

Supports flexible appearance attribute editing.

Abstract

Human video motion transfer (HVMT) aims to synthesize videos that one person imitates other persons' actions. Although existing GAN-based HVMT methods have achieved great success, they either fail to preserve appearance details due to the loss of spatial consistency between synthesized and exemplary images, or generate incoherent video results due to the lack of temporal consistency among video frames. In this paper, we propose Coarse-to-Fine Flow Warping Network (C2F-FWN) for spatial-temporal consistent HVMT. Particularly, C2F-FWN utilizes coarse-to-fine flow warping and Layout-Constrained Deformable Convolution (LC-DConv) to improve spatial consistency, and employs Flow Temporal Consistency (FTC) Loss to enhance temporal consistency. In addition, provided with multi-source appearance inputs, C2F-FWN can support appearance attribute editing with great flexibility and efficiency. Besides public datasets, we also collected a large-scale HVMT dataset named SoloDance for evaluation. Extensive experiments conducted on our SoloDance dataset and the iPER dataset show that our approach outperforms state-of-art HVMT methods in terms of both spatial and temporal consistency. Source code and the SoloDance dataset are available at https://github.com/wswdx/C2F-FWN.