Zero-shot Reconstruction of In-Scene Object Manipulation from Video

TL;DR

This paper introduces a novel system that reconstructs in-scene object manipulation from monocular RGB videos, leveraging foundation models and a two-stage optimization to recover detailed hand-object interactions.

Contribution

It is the first to combine data-driven initialization with optimization for accurate, physically plausible in-scene object manipulation reconstruction from monocular video.

Findings

Achieves detailed hand-object motion reconstruction from monocular videos.

Outperforms existing methods by incorporating scene context.

Provides a complete pipeline from initialization to optimization.

Abstract

We build the first system to address the problem of reconstructing in-scene object manipulation from a monocular RGB video. It is challenging due to ill-posed scene reconstruction, ambiguous hand-object depth, and the need for physically plausible interactions. Existing methods operate in hand centric coordinates and ignore the scene, hindering metric accuracy and practical use. In our method, we first use data-driven foundation models to initialize the core components, including the object mesh and poses, the scene point cloud, and the hand poses. We then apply a two-stage optimization that recovers a complete hand-object motion from grasping to interaction, which remains consistent with the scene information observed in the input video.