HandOS: 3D Hand Reconstruction in One Stage

TL;DR

HandOS introduces an end-to-end one-stage framework for 3D hand reconstruction that integrates detection, 2D and 3D keypoint estimation, reducing computation and errors compared to multi-stage methods.

Contribution

The paper presents a novel end-to-end framework that combines hand detection, 2D pose estimation, and 3D mesh reconstruction in a single stage, eliminating the need for multi-stage processing.

Findings

Achieves state-of-the-art performance on FreiHand with 5.0 PA-MPJPE.

Attains 64.6% PCK@0.05 on HInt-Ego4D benchmark.

Reduces computational redundancy and error accumulation.

Abstract

Existing approaches of hand reconstruction predominantly adhere to a multi-stage framework, encompassing detection, left-right classification, and pose estimation. This paradigm induces redundant computation and cumulative errors. In this work, we propose HandOS, an end-to-end framework for 3D hand reconstruction. Our central motivation lies in leveraging a frozen detector as the foundation while incorporating auxiliary modules for 2D and 3D keypoint estimation. In this manner, we integrate the pose estimation capacity into the detection framework, while at the same time obviating the necessity of using the left-right category as a prerequisite. Specifically, we propose an interactive 2D-3D decoder, where 2D joint semantics is derived from detection cues while 3D representation is lifted from those of 2D joints. Furthermore, hierarchical attention is designed to enable the concurrent modeling of 2D joints, 3D vertices, and camera translation. Consequently, we achieve an end-to-end integration of hand detection, 2D pose estimation, and 3D mesh reconstruction within a one-stage framework, so that the above multi-stage drawbacks are overcome. Meanwhile, the HandOS reaches state-of-the-art performances on public benchmarks, e.g., 5.0 PA-MPJPE on FreiHand and 64.6\% PCK@0.05 on HInt-Ego4D. Project page: idea-research.github.io/HandOSweb.