InterMimic: Towards Universal Whole-Body Control for Physics-Based Human-Object Interactions

TL;DR

InterMimic introduces a curriculum-based framework enabling a single policy to learn realistic, diverse, and generalizable human-object interactions from imperfect motion capture data, surpassing simple imitation through reinforcement learning.

Contribution

The paper presents a novel curriculum strategy and distillation approach for training a universal policy for complex human-object interactions from imperfect data.

Findings

Produces realistic and diverse HOIs across datasets

Generalizes zero-shot to new interactions

Integrates with kinematic generators for generative modeling

Abstract

Achieving realistic simulations of humans interacting with a wide range of objects has long been a fundamental goal. Extending physics-based motion imitation to complex human-object interactions (HOIs) is challenging due to intricate human-object coupling, variability in object geometries, and artifacts in motion capture data, such as inaccurate contacts and limited hand detail. We introduce InterMimic, a framework that enables a single policy to robustly learn from hours of imperfect MoCap data covering diverse full-body interactions with dynamic and varied objects. Our key insight is to employ a curriculum strategy -- perfect first, then scale up. We first train subject-specific teacher policies to mimic, retarget, and refine motion capture data. Next, we distill these teachers into a student policy, with the teachers acting as online experts providing direct supervision, as well as high-quality references. Notably, we incorporate RL fine-tuning on the student policy to surpass mere demonstration replication and achieve higher-quality solutions. Our experiments demonstrate that InterMimic produces realistic and diverse interactions across multiple HOI datasets. The learned policy generalizes in a zero-shot manner and seamlessly integrates with kinematic generators, elevating the framework from mere imitation to generative modeling of complex human-object interactions.