Pi-HOC: Pairwise 3D Human-Object Contact Estimation

TL;DR

Pi-HOC is a novel framework that accurately predicts dense 3D human-object contacts in images, handling multiple humans and objects efficiently, and improves 3D reconstruction and language-based contact queries.

Contribution

Introduces Pi-HOC, a single-pass, instance-aware model for dense 3D human-object contact prediction that outperforms existing methods in accuracy and speed.

Findings

Significantly improves contact prediction accuracy on MMHOI and DAMON datasets.

Achieves 20x higher inference throughput compared to previous methods.

Enhances 3D reconstruction and language-based contact understanding using predicted contacts.

Abstract

Resolving real-world human-object interactions in images is a many-to-many challenge, in which disentangling fine-grained concurrent physical contact is particularly difficult. Existing semantic contact estimation methods are either limited to single-human settings or require object geometries (e.g., meshes) in addition to the input image. Current state-of-the-art leverages powerful VLM for category-level semantics but struggles with multi-human scenarios and scales poorly in inference. We introduce Pi-HOC, a single-pass, instance-aware framework for dense 3D semantic contact prediction of all human-object pairs. Pi-HOC detects instances, creates dedicated human-object (HO) tokens for each pair, and refines them using an InteractionFormer. A SAM-based decoder then predicts dense contact on SMPL human meshes for each human-object pair. On the MMHOI and DAMON datasets, Pi-HOC significantly improves accuracy and localization over state-of-the-art methods while achieving 20x higher throughput. We further demonstrate that predicted contacts improve SAM-3D image-to-mesh reconstruction via a test-time optimization algorithm and enable referential contact prediction from language queries without additional training.