MimicMotion: High-Quality Human Motion Video Generation with Confidence-aware Pose Guidance

TL;DR

MimicMotion is a controllable video generation framework that produces high-quality, long, and smooth human motion videos by leveraging confidence-aware pose guidance and a progressive latent fusion strategy.

Contribution

The paper introduces confidence-aware pose guidance, regional loss amplification, and a progressive latent fusion strategy for high-quality, controllable, and long human motion video generation.

Findings

Significant quality improvements over previous methods.

Effective generation of arbitrarily long videos.

Enhanced temporal smoothness and detail richness.

Abstract

In recent years, generative artificial intelligence has achieved significant advancements in the field of image generation, spawning a variety of applications. However, video generation still faces considerable challenges in various aspects, such as controllability, video length, and richness of details, which hinder the application and popularization of this technology. In this work, we propose a controllable video generation framework, dubbed MimicMotion, which can generate high-quality videos of arbitrary length mimicking specific motion guidance. Compared with previous methods, our approach has several highlights. Firstly, we introduce confidence-aware pose guidance that ensures high frame quality and temporal smoothness. Secondly, we introduce regional loss amplification based on pose confidence, which significantly reduces image distortion. Lastly, for generating long and smooth videos, we propose a progressive latent fusion strategy. By this means, we can produce videos of arbitrary length with acceptable resource consumption. With extensive experiments and user studies, MimicMotion demonstrates significant improvements over previous approaches in various aspects. Detailed results and comparisons are available on our project page: https://tencent.github.io/MimicMotion .