Interacting Hand-Object Pose Estimation via Dense Mutual Attention

TL;DR

This paper introduces a dense mutual attention mechanism for 3D hand-object pose estimation, enabling fine-grained modeling of interactions that leads to more accurate and real-time results.

Contribution

It proposes a novel dense mutual attention approach that models detailed hand-object dependencies using graph structures, improving accuracy and efficiency.

Findings

Outperforms state-of-the-art methods on benchmark datasets

Achieves high-quality, physically plausible pose estimations

Enables real-time inference speeds

Abstract

3D hand-object pose estimation is the key to the success of many computer vision applications. The main focus of this task is to effectively model the interaction between the hand and an object. To this end, existing works either rely on interaction constraints in a computationally-expensive iterative optimization, or consider only a sparse correlation between sampled hand and object keypoints. In contrast, we propose a novel dense mutual attention mechanism that is able to model fine-grained dependencies between the hand and the object. Specifically, we first construct the hand and object graphs according to their mesh structures. For each hand node, we aggregate features from every object node by the learned attention and vice versa for each object node. Thanks to such dense mutual attention, our method is able to produce physically plausible poses with high quality and real-time inference speed. Extensive quantitative and qualitative experiments on large benchmark datasets show that our method outperforms state-of-the-art methods. The code is available at https://github.com/rongakowang/DenseMutualAttention.git.