HandMIM: Pose-Aware Self-Supervised Learning for 3D Hand Mesh Estimation

TL;DR

HandMIM introduces a pose-aware self-supervised learning method for 3D hand mesh estimation, leveraging Masked Image Modeling and a teacher-student framework to improve accuracy without relying on detection models.

Contribution

The paper proposes a novel self-supervised pre-training strategy with pose-aware modules and multi-level learning for 3D hand mesh estimation, outperforming existing architectures.

Findings

Achieves 6.29mm PAVPE on FreiHAND

Achieves 8.00mm PAVPE on HO3Dv2

Sets new state-of-the-art results in 3D hand mesh estimation

Abstract

With an enormous number of hand images generated over time, unleashing pose knowledge from unlabeled images for supervised hand mesh estimation is an emerging yet challenging topic. To alleviate this issue, semi-supervised and self-supervised approaches have been proposed, but they are limited by the reliance on detection models or conventional ResNet backbones. In this paper, inspired by the rapid progress of Masked Image Modeling (MIM) in visual classification tasks, we propose a novel self-supervised pre-training strategy for regressing 3D hand mesh parameters. Our approach involves a unified and multi-granularity strategy that includes a pseudo keypoint alignment module in the teacher-student framework for learning pose-aware semantic class tokens. For patch tokens with detailed locality, we adopt a self-distillation manner between teacher and student network based on MIM pre-training. To better fit low-level regression tasks, we incorporate pixel reconstruction tasks for multi-level representation learning. Additionally, we design a strong pose estimation baseline using a simple vanilla vision Transformer (ViT) as the backbone and attach a PyMAF head after tokens for regression. Extensive experiments demonstrate that our proposed approach, named HandMIM, achieves strong performance on various hand mesh estimation tasks. Notably, HandMIM outperforms specially optimized architectures, achieving 6.29mm and 8.00mm PAVPE (Vertex-Point-Error) on challenging FreiHAND and HO3Dv2 test sets, respectively, establishing new state-of-the-art records on 3D hand mesh estimation.