Skeleton2Humanoid: Animating Simulated Characters for Physically-plausible Motion In-betweening

TL;DR

Skeleton2Humanoid is a physics-based system that enhances human motion synthesis by ensuring physical plausibility through test time adaptation, inverse kinematics, and reinforcement learning, significantly improving realism and accuracy.

Contribution

The paper introduces a novel three-stage framework combining test time adaptation, inverse kinematics, and RL to produce physically plausible human motions from skeleton data.

Findings

Outperforms prior methods in physical plausibility and accuracy

Effective motion correction via physics simulation and RL

Improved motion-in-betweening results on LaFAN1 dataset

Abstract

Human motion synthesis is a long-standing problem with various applications in digital twins and the Metaverse. However, modern deep learning based motion synthesis approaches barely consider the physical plausibility of synthesized motions and consequently they usually produce unrealistic human motions. In order to solve this problem, we propose a system ``Skeleton2Humanoid'' which performs physics-oriented motion correction at test time by regularizing synthesized skeleton motions in a physics simulator. Concretely, our system consists of three sequential stages: (I) test time motion synthesis network adaptation, (II) skeleton to humanoid matching and (III) motion imitation based on reinforcement learning (RL). Stage I introduces a test time adaptation strategy, which improves the physical plausibility of synthesized human skeleton motions by optimizing skeleton joint locations. Stage II performs an analytical inverse kinematics strategy, which converts the optimized human skeleton motions to humanoid robot motions in a physics simulator, then the converted humanoid robot motions can be served as reference motions for the RL policy to imitate. Stage III introduces a curriculum residual force control policy, which drives the humanoid robot to mimic complex converted reference motions in accordance with the physical law. We verify our system on a typical human motion synthesis task, motion-in-betweening. Experiments on the challenging LaFAN1 dataset show our system can outperform prior methods significantly in terms of both physical plausibility and accuracy. Code will be released for research purposes at: https://github.com/michaelliyunhao/Skeleton2Humanoid