InterMesh: Explicit Interaction-Aware End-to-End Multi-Person Human Mesh Recovery

TL;DR

InterMesh introduces an explicit interaction-aware framework for multi-person human mesh recovery, significantly improving pose estimation accuracy by incorporating human-object interaction information.

Contribution

It explicitly models human-environment interactions using a human-object interaction detector, enhancing existing human mesh recovery architectures with minimal overhead.

Findings

Reduces MPJPE by 9.9% on CMU Panoptic dataset.

Achieves an 8.2% reduction in MPJPE on Hi4D.

Demonstrates superior performance over state-of-the-art methods on multiple datasets.

Abstract

Humans constantly interact with their surroundings. Existing end-to-end multi-person human mesh recovery methods, typically based on the DETR framework, capture inter-human relationships through self-attention across all human queries. However, these approaches model interactions only implicitly and lack explicit reasoning about how humans interact with objects and with each other. In this paper, we propose InterMesh, a simple yet effective framework that explicitly incorporates human-environment interaction information into human mesh recovery pipeline. By leveraging a human-object interaction detector, InterMesh enriches query representations with structured interaction semantics, enabling more accurate pose and shape estimation. We design lightweight modules, Contextual Interaction Encoder and Interaction-Guided Refiner, to integrate these features into existing HMR architectures with minimal overhead. We validate our approach through extensive experiments on 3DPW, MuPoTS, CMU Panoptic, Hi4D, and CHI3D datasets, demonstrating remarkable improvements over state-of-the-art methods. Notably, InterMesh reduces MPJPE by 9.9% on CMU Panoptic and 8.2% on Hi4D, highlighting its effectiveness in scenarios with complex human-object and inter-human interactions. Code and models are released at https://github.com/Kelly510/InterMesh.