STMR: Spiral Transformer for Hand Mesh Reconstruction

TL;DR

This paper introduces STMR, a novel spiral transformer architecture for hand mesh reconstruction that integrates spiral sampling with transformers, employing multi-scale pose features and pose-to-vertex lifting to achieve state-of-the-art accuracy and speed.

Contribution

The paper presents a new spiral transformer model that effectively leverages mesh topology for hand reconstruction, incorporating MSPFE and PPVL modules for enhanced feature extraction.

Findings

Achieves state-of-the-art accuracy on FreiHAND dataset

Demonstrates superior inference speed compared to similar methods

Shows improved mesh reconstruction performance with novel modules

Abstract

Recent advancements in both transformer-based methods and spiral neighbor sampling techniques have greatly enhanced hand mesh reconstruction. Transformers excel in capturing complex vertex relationships, and spiral neighbor sampling is vital for utilizing topological structures. This paper ingeniously integrates spiral sampling into the Transformer architecture, enhancing its ability to leverage mesh topology for superior performance in hand mesh reconstruction, resulting in substantial accuracy boosts. STMR employs a single image encoder for model efficiency. To augment its information extraction capability, we design the multi-scale pose feature extraction (MSPFE) module, which facilitates the extraction of rich pose features, ultimately enhancing the model's performance. Moreover, the proposed predefined pose-to-vertex lifting (PPVL) method improves vertex feature representation, further boosting reconstruction performance. Extensive experiments on the FreiHAND dataset demonstrate the state-of-the-art performance and unparalleled inference speed of STMR compared with similar backbone methods, showcasing its efficiency and effectiveness. The code is available at https://github.com/SmallXieGithub/STMR.