Can Action be Imitated? Learn to Reconstruct and Transfer Human Dynamics from Videos

TL;DR

This paper introduces a novel mesh-based action imitation task that reconstructs and transfers human actions from videos to target meshes, enabling realistic and consistent human motion replication.

Contribution

We propose the M-VAI method that reconstructs human meshes from videos, improves temporal consistency with mesh2mesh, and transfers actions to target identities, advancing human motion imitation.

Findings

High-quality human mesh generation demonstrated

Effective action transfer to target meshes shown

Temporal consistency improved with mesh2mesh

Abstract

Given a video demonstration, can we imitate the action contained in this video? In this paper, we introduce a novel task, dubbed mesh-based action imitation. The goal of this task is to enable an arbitrary target human mesh to perform the same action shown on the video demonstration. To achieve this, a novel Mesh-based Video Action Imitation (M-VAI) method is proposed by us. M-VAI first learns to reconstruct the meshes from the given source image frames, then the initial recovered mesh sequence is fed into mesh2mesh, a mesh sequence smooth module proposed by us, to improve the temporal consistency. Finally, we imitate the actions by transferring the pose from the constructed human body to our target identity mesh. High-quality and detailed human body meshes can be generated by using our M-VAI. Extensive experiments demonstrate the feasibility of our task and the effectiveness of our proposed method.