SiMA-Hand: Boosting 3D Hand-Mesh Reconstruction by Single-to-Multi-View Adaptation

TL;DR

SiMA-Hand introduces a novel single-to-multi-view adaptation approach that significantly improves 3D hand mesh reconstruction from single RGB images by leveraging multi-view training to handle occlusions effectively.

Contribution

The paper proposes a multi-view fusion framework and a single-view reconstructor with SiMA to enhance occlusion handling in 3D hand mesh estimation.

Findings

Outperforms state-of-the-art on Dex-YCB and HanCo benchmarks.

Effectively handles occlusion in challenging scenarios.

Enables single-view inference with enriched features learned from multi-view data.

Abstract

Estimating 3D hand mesh from RGB images is a longstanding track, in which occlusion is one of the most challenging problems. Existing attempts towards this task often fail when the occlusion dominates the image space. In this paper, we propose SiMA-Hand, aiming to boost the mesh reconstruction performance by Single-to-Multi-view Adaptation. First, we design a multi-view hand reconstructor to fuse information across multiple views by holistically adopting feature fusion at image, joint, and vertex levels. Then, we introduce a single-view hand reconstructor equipped with SiMA. Though taking only one view as input at inference, the shape and orientation features in the single-view reconstructor can be enriched by learning non-occluded knowledge from the extra views at training, enhancing the reconstruction precision on the occluded regions. We conduct experiments on the Dex-YCB and HanCo benchmarks with challenging object- and self-caused occlusion cases, manifesting that SiMA-Hand consistently achieves superior performance over the state of the arts. Code will be released on https://github.com/JoyboyWang/SiMA-Hand Pytorch.