OmniHands: Towards Robust 4D Hand Mesh Recovery via A Versatile Transformer

TL;DR

OmniHands introduces a versatile transformer-based method for robust 4D hand mesh recovery from monocular and multi-view inputs, effectively modeling hand interactions and relative movements.

Contribution

It presents a universal architecture with novel tokenization and feature fusion strategies to handle various hand input types and relative hand positions.

Findings

Achieves superior performance on benchmark datasets.

Effectively reconstructs intricate hand interactions in real-world scenarios.

Handles both single-hand and two-hand inputs with explicit relative positioning.

Abstract

In this paper, we introduce OmniHands, a universal approach to recovering interactive hand meshes and their relative movement from monocular or multi-view inputs. Our approach addresses two major limitations of previous methods: lacking a unified solution for handling various hand image inputs and neglecting the positional relationship of two hands within images. To overcome these challenges, we develop a universal architecture with novel tokenization and contextual feature fusion strategies, capable of adapting to a variety of tasks. Specifically, we propose a Relation-aware Two-Hand Tokenization (RAT) method to embed positional relation information into the hand tokens. In this way, our network can handle both single-hand and two-hand inputs and explicitly leverage relative hand positions, facilitating the reconstruction of intricate hand interactions in real-world scenarios. As such tokenization indicates the relative relationship of two hands, it also supports more effective feature fusion. To this end, we further develop a 4D Interaction Reasoning (FIR) module to fuse hand tokens in 4D with attention and decode them into 3D hand meshes and relative temporal movements. The efficacy of our approach is validated on several benchmark datasets. The results on in-the-wild videos and real-world scenarios demonstrate the superior performances of our approach for interactive hand reconstruction. More video results can be found on the project page: https://OmniHand.github.io.