ShapeGraFormer: GraFormer-Based Network for Hand-Object Reconstruction from a Single Depth Map

TL;DR

ShapeGraFormer introduces a novel GraFormer-based network that accurately reconstructs 3D hand-object shapes and poses from a single depth map, emphasizing realistic interactions and shape refinement.

Contribution

It is the first to use GraFormer for detailed 3D hand-object shape and pose reconstruction from a single depth image, incorporating shape refinement based on interactions.

Findings

Outperforms existing methods in hand reconstruction accuracy

Produces plausible and detailed object shapes in interactions

Effective shape refinement improves reconstruction realism

Abstract

3D reconstruction of hand-object manipulations is important for emulating human actions. Most methods dealing with challenging object manipulation scenarios, focus on hands reconstruction in isolation, ignoring physical and kinematic constraints due to object contact. Some approaches produce more realistic results by jointly reconstructing 3D hand-object interactions. However, they focus on coarse pose estimation or rely upon known hand and object shapes. We propose the first approach for realistic 3D hand-object shape and pose reconstruction from a single depth map. Unlike previous work, our voxel-based reconstruction network regresses the vertex coordinates of a hand and an object and reconstructs more realistic interaction. Our pipeline additionally predicts voxelized hand-object shapes, having a one-to-one mapping to the input voxelized depth. Thereafter, we exploit the graph nature of the hand and object shapes, by utilizing the recent GraFormer network with positional embedding to reconstruct shapes from template meshes. In addition, we show the impact of adding another GraFormer component that refines the reconstructed shapes based on the hand-object interactions and its ability to reconstruct more accurate object shapes. We perform an extensive evaluation on the HO-3D and DexYCB datasets and show that our method outperforms existing approaches in hand reconstruction and produces plausible reconstructions for the objects