Residual Force Control for Agile Human Behavior Imitation and Extended Motion Synthesis

TL;DR

This paper introduces residual force control (RFC), a novel method that enhances humanoid motion synthesis by compensating for dynamics mismatch, enabling realistic, agile, and long-term human behavior imitation including complex motions like ballet.

Contribution

The paper proposes RFC, a new approach that adds residual forces to improve humanoid control policies, allowing for more realistic and diverse long-term motion synthesis from large-scale datasets.

Findings

RFC outperforms state-of-the-art methods in motion quality and convergence speed.

The approach enables humanoids to perform complex ballet moves like pirouettes and arabesques.

First humanoid control method to learn from large-scale human motion datasets for diverse long-term motions.

Abstract

Reinforcement learning has shown great promise for synthesizing realistic human behaviors by learning humanoid control policies from motion capture data. However, it is still very challenging to reproduce sophisticated human skills like ballet dance, or to stably imitate long-term human behaviors with complex transitions. The main difficulty lies in the dynamics mismatch between the humanoid model and real humans. That is, motions of real humans may not be physically possible for the humanoid model. To overcome the dynamics mismatch, we propose a novel approach, residual force control (RFC), that augments a humanoid control policy by adding external residual forces into the action space. During training, the RFC-based policy learns to apply residual forces to the humanoid to compensate for the dynamics mismatch and better imitate the reference motion. Experiments on a wide range of dynamic motions demonstrate that our approach outperforms state-of-the-art methods in terms of convergence speed and the quality of learned motions. Notably, we showcase a physics-based virtual character empowered by RFC that can perform highly agile ballet dance moves such as pirouette, arabesque and jet\'e. Furthermore, we propose a dual-policy control framework, where a kinematic policy and an RFC-based policy work in tandem to synthesize multi-modal infinite-horizon human motions without any task guidance or user input. Our approach is the first humanoid control method that successfully learns from a large-scale human motion dataset (Human3.6M) and generates diverse long-term motions. Code and videos are available at https://www.ye-yuan.com/rfc.