Delving into the Frequency: Temporally Consistent Human Motion Transfer in the Fourier Space

TL;DR

This paper investigates the frequency domain differences between natural and synthetic videos for human motion transfer, proposing a novel frequency-based framework that enhances temporal consistency and visual quality in synthesized videos.

Contribution

The paper introduces the first comprehensive frequency domain analysis for human motion transfer and proposes FreMOTR, a framework with two novel regularization modules to improve temporal and spatial consistency.

Findings

Nearly 30% improvement in temporal consistency metrics.

Enhanced frame-level visual quality of synthetic videos.

Effective mitigation of spatial artifacts and temporal inconsistency.

Abstract

Human motion transfer refers to synthesizing photo-realistic and temporally coherent videos that enable one person to imitate the motion of others. However, current synthetic videos suffer from the temporal inconsistency in sequential frames that significantly degrades the video quality, yet is far from solved by existing methods in the pixel domain. Recently, some works on DeepFake detection try to distinguish the natural and synthetic images in the frequency domain because of the frequency insufficiency of image synthesizing methods. Nonetheless, there is no work to study the temporal inconsistency of synthetic videos from the aspects of the frequency-domain gap between natural and synthetic videos. In this paper, we propose to delve into the frequency space for temporally consistent human motion transfer. First of all, we make the first comprehensive analysis of natural and synthetic videos in the frequency domain to reveal the frequency gap in both the spatial dimension of individual frames and the temporal dimension of the video. To close the frequency gap between the natural and synthetic videos, we propose a novel Frequency-based human MOtion TRansfer framework, named FreMOTR, which can effectively mitigate the spatial artifacts and the temporal inconsistency of the synthesized videos. FreMOTR explores two novel frequency-based regularization modules: 1) the Frequency-domain Appearance Regularization (FAR) to improve the appearance of the person in individual frames and 2) Temporal Frequency Regularization (TFR) to guarantee the temporal consistency between adjacent frames. Finally, comprehensive experiments demonstrate that the FreMOTR not only yields superior performance in temporal consistency metrics but also improves the frame-level visual quality of synthetic videos. In particular, the temporal consistency metrics are improved by nearly 30% than the state-of-the-art model.