CRISP: Contact-Guided Real2Sim from Monocular Video with Planar Scene Primitives

TL;DR

CRISP is a novel method that reconstructs clean, convex scene geometry and human motion from monocular video, enabling more reliable physics-based simulation for robotics and AR/VR applications.

Contribution

CRISP introduces a contact-guided approach that fits planar primitives to scene point clouds and uses human contact modeling to improve scene reconstruction and motion simulation accuracy.

Findings

Reduced motion tracking failure rate from 55.2% to 6.9%.

Increased RL simulation throughput by 43%.

Validated on diverse in-the-wild videos.

Abstract

We introduce CRISP, a method that recovers simulatable human motion and scene geometry from monocular video. Prior work on joint human-scene reconstruction relies on data-driven priors and joint optimization with no physics in the loop, or recovers noisy geometry with artifacts that cause motion tracking policies with scene interactions to fail. In contrast, our key insight is to recover convex, clean, and simulation-ready geometry by fitting planar primitives to a point cloud reconstruction of the scene, via a simple clustering pipeline over depth, normals, and flow. To reconstruct scene geometry that might be occluded during interactions, we make use of human-scene contact modeling (e.g., we use human posture to reconstruct the occluded seat of a chair). Finally, we ensure that human and scene reconstructions are physically-plausible by using them to drive a humanoid controller via reinforcement learning. Our approach reduces motion tracking failure rates from 55.2\% to 6.9\% on human-centric video benchmarks (EMDB, PROX), while delivering a 43\% faster RL simulation throughput. We further validate it on in-the-wild videos including casually-captured videos, Internet videos, and even Sora-generated videos. This demonstrates CRISP's ability to generate physically-valid human motion and interaction environments at scale, greatly advancing real-to-sim applications for robotics and AR/VR.