ParaHome: Parameterizing Everyday Home Activities Towards 3D Generative Modeling of Human-Object Interactions

TL;DR

ParaHome introduces a comprehensive multi-view and wearable sensor system to capture detailed 3D human-object interactions in home environments, creating a rich dataset for advancing generative modeling of daily activities.

Contribution

The paper presents a novel multi-view and wearable sensor system for capturing complex 3D human-object interactions in home settings, along with a new dataset for generative modeling.

Findings

Captured 486 minutes of interaction data with 38 participants.

Enabled modeling of body motion, hand manipulation, and articulated objects.

Demonstrated potential for generative modeling experiments.

Abstract

To enable machines to understand the way humans interact with the physical world in daily life, 3D interaction signals should be captured in natural settings, allowing people to engage with multiple objects in a range of sequential and casual manipulations. To achieve this goal, we introduce our ParaHome system designed to capture dynamic 3D movements of humans and objects within a common home environment. Our system features a multi-view setup with 70 synchronized RGB cameras, along with wearable motion capture devices including an IMU-based body suit and hand motion capture gloves. By leveraging the ParaHome system, we collect a new human-object interaction dataset, including 486 minutes of sequences across 207 captures with 38 participants, offering advancements with three key aspects: (1) capturing body motion and dexterous hand manipulation motion alongside multiple objects within a contextual home environment; (2) encompassing sequential and concurrent manipulations paired with text descriptions; and (3) including articulated objects with multiple parts represented by 3D parameterized models. We present detailed design justifications for our system, and perform key generative modeling experiments to demonstrate the potential of our dataset.