Extract-and-Adaptation Network for 3D Interacting Hand Mesh Recovery

TL;DR

This paper introduces EANet, a novel network that improves 3D interacting hand mesh recovery by using specialized tokens to better model hand interactions, overcoming limitations of previous Transformer-based methods.

Contribution

The paper proposes EABlock with SimToken and JoinToken to enhance interaction modeling and address the distant token problem in Transformer-based hand mesh recovery.

Findings

Achieves state-of-the-art performance on 3D interacting hands benchmarks.

Effectively models hand interactions with novel token design.

Robust to different hand pose configurations.

Abstract

Understanding how two hands interact with each other is a key component of accurate 3D interacting hand mesh recovery. However, recent Transformer-based methods struggle to learn the interaction between two hands as they directly utilize two hand features as input tokens, which results in distant token problem. The distant token problem represents that input tokens are in heterogeneous spaces, leading Transformer to fail in capturing correlation between input tokens. Previous Transformer-based methods suffer from the problem especially when poses of two hands are very different as they project features from a backbone to separate left and right hand-dedicated features. We present EANet, extract-and-adaptation network, with EABlock, the main component of our network. Rather than directly utilizing two hand features as input tokens, our EABlock utilizes two complementary types of novel tokens, SimToken and JoinToken, as input tokens. Our two novel tokens are from a combination of separated two hand features; hence, it is much more robust to the distant token problem. Using the two type of tokens, our EABlock effectively extracts interaction feature and adapts it to each hand. The proposed EANet achieves the state-of-the-art performance on 3D interacting hands benchmarks. The codes are available at https://github.com/jkpark0825/EANet.