Reconstructing Three-Dimensional Models of Interacting Humans

TL;DR

This paper introduces new models and datasets for accurately reconstructing 3D human interactions, emphasizing contact detection and consistency to improve behavioral understanding.

Contribution

It presents models for interaction signature estimation, datasets for contact prediction, and methods for reconstructing and annotating 3D interactions in controlled setups.

Findings

Developed CHI3D dataset with 631 sequences and 2,525 contact events.

Created FlickrCI3D dataset with over 11,000 images and 81,000 surface correspondences.

Proposed methodology for recovering ground-truth pose and shape of interacting humans.

Abstract

Understanding 3d human interactions is fundamental for fine-grained scene analysis and behavioural modeling. However, most of the existing models predict incorrect, lifeless 3d estimates, that miss the subtle human contact aspects--the essence of the event--and are of little use for detailed behavioral understanding. This paper addresses such issues with several contributions: (1) we introduce models for interaction signature estimation (ISP) encompassing contact detection, segmentation, and 3d contact signature prediction; (2) we show how such components can be leveraged to ensure contact consistency during 3d reconstruction; (3) we construct several large datasets for learning and evaluating 3d contact prediction and reconstruction methods; specifically, we introduce CHI3D, a lab-based accurate 3d motion capture dataset with 631 sequences containing $2,525$ contact events, $728,664$ ground truth 3d poses, as well as FlickrCI3D, a dataset of $11,216$ images, with $14,081$ processed pairs of people, and $81,233$ facet-level surface correspondences. Finally, (4) we propose methodology for recovering the ground-truth pose and shape of interacting people in a controlled setup and (5) annotate all 3d interaction motions in CHI3D with textual descriptions. Motion data in multiple formats (GHUM and SMPLX parameters, Human3.6m 3d joints) is made available for research purposes at \url{https://ci3d.imar.ro}, together with an evaluation server and a public benchmark.