Interaction Replica: Tracking Human-Object Interaction and Scene Changes From Human Motion

TL;DR

This paper introduces iReplica, a novel method for modeling human-object interactions and scene changes using simple ego-centric sensors, enabling realistic prediction of interactions without external tracking systems.

Contribution

We develop a new approach combining visual localization and contact-based reasoning from IMU data for egocentric interaction modeling, filling a gap in current research.

Findings

Human-scene contacts can be predicted from pose sequences.

The method enables realistic interaction modeling without external cameras.

iReplica advances egocentric capture for AR/VR and robotics applications.

Abstract

Our world is not static and humans naturally cause changes in their environments through interactions, e.g., opening doors or moving furniture. Modeling changes caused by humans is essential for building digital twins, e.g., in the context of shared physical-virtual spaces (metaverses) and robotics. In order for widespread adoption of such emerging applications, the sensor setup used to capture the interactions needs to be inexpensive and easy-to-use for non-expert users. I.e., interactions should be captured and modeled by simple ego-centric sensors such as a combination of cameras and IMU sensors, not relying on any external cameras or object trackers. Yet, to the best of our knowledge, no work tackling the challenging problem of modeling human-scene interactions via such an ego-centric sensor setup exists. This paper closes this gap in the literature by developing a novel approach that combines visual localization of humans in the scene with contact-based reasoning about human-scene interactions from IMU data. Interestingly, we can show that even without visual observations of the interactions, human-scene contacts and interactions can be realistically predicted from human pose sequences. Our method, iReplica (Interaction Replica), is an essential first step towards the egocentric capture of human interactions and modeling of dynamic scenes, which is required for future AR/VR applications in immersive virtual universes and for training machines to behave like humans. Our code, data and model are available on our project page at http://virtualhumans.mpi-inf.mpg.de/ireplica/