Learning to Control Physically-simulated 3D Characters via Generating and Mimicking 2D Motions

TL;DR

This paper introduces Mimic2DM, a framework that learns to control 3D characters in physics simulations by generating and mimicking 2D motions extracted from videos, eliminating the need for 3D motion data.

Contribution

It proposes a novel approach that trains a 2D motion tracking policy directly from 2D keypoints and integrates a transformer-based 2D motion generator for diverse, physically plausible 3D character motions.

Findings

Effective synthesis of diverse motions like dancing, soccer, and animal movements.

No reliance on explicit 3D motion data or reconstruction techniques.

Versatile framework applicable across multiple domains.

Abstract

Video data is more cost-effective than motion capture data for learning 3D character motion controllers, yet synthesizing realistic and diverse behaviors directly from videos remains challenging. Previous approaches typically rely on off-the-shelf motion reconstruction techniques to obtain 3D trajectories for physics-based imitation. These reconstruction methods struggle with generalizability, as they either require 3D training data (potentially scarce) or fail to produce physically plausible poses, hindering their application to challenging scenarios like human-object interaction (HOI) or non-human characters. We tackle this challenge by introducing Mimic2DM, a novel motion imitation framework that learns the control policy directly and solely from widely available 2D keypoint trajectories extracted from videos. By minimizing the reprojection error, we train a general single-view 2D motion tracking policy capable of following arbitrary 2D reference motions in physics simulation, using only 2D motion data. The policy, when trained on diverse 2D motions captured from different or slightly different viewpoints, can further acquire 3D motion tracking capabilities by aggregating multiple views. Moreover, we develop a transformer-based autoregressive 2D motion generator and integrate it into a hierarchical control framework, where the generator produces high-quality 2D reference trajectories to guide the tracking policy. We show that the proposed approach is versatile and can effectively learn to synthesize physically plausible and diverse motions across a range of domains, including dancing, soccer dribbling, and animal movements, without any reliance on explicit 3D motion data. Project Website: https://jiann-li.github.io/mimic2dm/