Kinematics-based 3D Human-Object Interaction Reconstruction from Single View

TL;DR

This paper introduces a kinematics-based approach for reconstructing 3D human-object interactions from single-view images, leveraging improved forward and inverse kinematics and a contact region recognition network to enhance accuracy and robustness.

Contribution

The paper presents a novel kinematics-based method with an improved forward kinematics algorithm and a neural inverse kinematics solution, plus a contact region recognition network, for better 3D HOI reconstruction from single views.

Findings

Outperforms state-of-the-art on BEHAVE benchmark.

Achieves more accurate joint poses than traditional robotics methods.

Demonstrates good portability and integration capability.

Abstract

Reconstructing 3D human-object interaction (HOI) from single-view RGB images is challenging due to the absence of depth information and potential occlusions. Existing methods simply predict the body poses merely rely on network training on some indoor datasets, which cannot guarantee the rationality of the results if some body parts are invisible due to occlusions that appear easily. Inspired by the end-effector localization task in robotics, we propose a kinematics-based method that can drive the joints of human body to the human-object contact regions accurately. After an improved forward kinematics algorithm is proposed, the Multi-Layer Perceptron is introduced into the solution of inverse kinematics process to determine the poses of joints, which achieves precise results than the commonly-used numerical methods in robotics. Besides, a Contact Region Recognition Network (CRRNet) is also proposed to robustly determine the contact regions using a single-view video. Experimental results demonstrate that our method outperforms the state-of-the-art on benchmark BEHAVE. Additionally, our approach shows good portability and can be seamlessly integrated into other methods for optimizations.