A Plug-and-Play Physical Motion Restoration Approach for In-the-Wild High-Difficulty Motions

TL;DR

This paper introduces a plug-and-play framework combining motion correction and physics-based transfer to improve the physical plausibility of high-difficulty in-the-wild 3D human motions extracted from videos, addressing flaws in existing methods.

Contribution

It presents a novel mask-based motion correction module and a physics-based motion transfer module that together enhance motion realism, especially for challenging in-the-wild motions, and establishes a new benchmark dataset.

Findings

Effective correction of flawed motions in challenging videos

Improved physical plausibility of high-difficulty motions

Validated on new and existing datasets

Abstract

Extracting physically plausible 3D human motion from videos is a critical task. Although existing simulation-based motion imitation methods can enhance the physical quality of daily motions estimated from monocular video capture, extending this capability to high-difficulty motions remains an open challenge. This can be attributed to some flawed motion clips in video-based motion capture results and the inherent complexity in modeling high-difficulty motions. Therefore, sensing the advantage of segmentation in localizing human body, we introduce a mask-based motion correction module (MCM) that leverages motion context and video mask to repair flawed motions, producing imitation-friendly motions; and propose a physics-based motion transfer module (PTM), which employs a pretrain and adapt approach for motion imitation, improving physical plausibility with the ability to handle in-the-wild and challenging motions. Our approach is designed as a plug-and-play module to physically refine the video motion capture results, including high-difficulty in-the-wild motions. Finally, to validate our approach, we collected a challenging in-the-wild test set to establish a benchmark, and our method has demonstrated effectiveness on both the new benchmark and existing public datasets.https://physicalmotionrestoration.github.io