Reconstructing Interacting Hands with Interaction Prior from Monocular Images

TL;DR

This paper introduces a novel approach for reconstructing interacting hands from monocular images by leveraging a learned interaction prior, a large-scale multimodal dataset, and a specialized heatmap, improving robustness against occlusion and similarity issues.

Contribution

It proposes a new interaction prior framework using a VAE and a multimodal dataset, along with the interaction adjacency heatmap for better feature extraction in hand interaction reconstruction.

Findings

Effective in handling occlusion and similarity challenges

Outperforms existing methods on benchmark datasets

Provides publicly available code and dataset

Abstract

Reconstructing interacting hands from monocular images is indispensable in AR/VR applications. Most existing solutions rely on the accurate localization of each skeleton joint. However, these methods tend to be unreliable due to the severe occlusion and confusing similarity among adjacent hand parts. This also defies human perception because humans can quickly imitate an interaction pattern without localizing all joints. Our key idea is to first construct a two-hand interaction prior and recast the interaction reconstruction task as the conditional sampling from the prior. To expand more interaction states, a large-scale multimodal dataset with physical plausibility is proposed. Then a VAE is trained to further condense these interaction patterns as latent codes in a prior distribution. When looking for image cues that contribute to interaction prior sampling, we propose the interaction adjacency heatmap (IAH). Compared with a joint-wise heatmap for localization, IAH assigns denser visible features to those invisible joints. Compared with an all-in-one visible heatmap, it provides more fine-grained local interaction information in each interaction region. Finally, the correlations between the extracted features and corresponding interaction codes are linked by the ViT module. Comprehensive evaluations on benchmark datasets have verified the effectiveness of this framework. The code and dataset are publicly available at https://github.com/binghui-z/InterPrior_pytorch