Reconstructing Close Human Interaction with Appearance and Proxemics Reasoning

TL;DR

This paper introduces a dual-branch optimization framework leveraging human appearance and proxemics to improve the reconstruction of close human interactions in challenging in-the-wild videos, surpassing existing methods.

Contribution

It proposes a novel framework that combines appearance cues, social proxemics, and physical constraints with a diffusion model to accurately reconstruct interactive human motions.

Findings

Outperforms existing pose estimation methods on several benchmarks.

Successfully reconstructs plausible close interactions in complex environments.

Builds a new dataset with pseudo ground-truth annotations for future research.

Abstract

Due to visual ambiguities and inter-person occlusions, existing human pose estimation methods cannot recover plausible close interactions from in-the-wild videos. Even state-of-the-art large foundation models~(\eg, SAM) cannot accurately distinguish human semantics in such challenging scenarios. In this work, we find that human appearance can provide a straightforward cue to address these obstacles. Based on this observation, we propose a dual-branch optimization framework to reconstruct accurate interactive motions with plausible body contacts constrained by human appearances, social proxemics, and physical laws. Specifically, we first train a diffusion model to learn the human proxemic behavior and pose prior knowledge. The trained network and two optimizable tensors are then incorporated into a dual-branch optimization framework to reconstruct human motions and appearances. Several constraints based on 3D Gaussians, 2D keypoints, and mesh penetrations are also designed to assist the optimization. With the proxemics prior and diverse constraints, our method is capable of estimating accurate interactions from in-the-wild videos captured in complex environments. We further build a dataset with pseudo ground-truth interaction annotations, which may promote future research on pose estimation and human behavior understanding. Experimental results on several benchmarks demonstrate that our method outperforms existing approaches. The code and data are available at https://www.buzhenhuang.com/works/CloseApp.html.