SHOWMe: Benchmarking Object-agnostic Hand-Object 3D Reconstruction

TL;DR

SHOWMe introduces a detailed 3D hand-object dataset and a pipeline for object-agnostic 3D reconstruction, enabling advancements in hand-object interaction analysis despite current limitations in initial pose estimation accuracy.

Contribution

The paper presents the SHOWMe dataset with detailed 3D textured meshes and a two-stage pipeline for reconstructing unknown hand-held objects, addressing limitations of previous datasets and methods.

Findings

Achieved promising 3D reconstructions using SfM and hand pose estimators.

Reconstruction methods are sensitive to initial camera pose estimates.

Rigid hand-object assumption enables effective 3D reconstruction despite challenges.

Abstract

Recent hand-object interaction datasets show limited real object variability and rely on fitting the MANO parametric model to obtain groundtruth hand shapes. To go beyond these limitations and spur further research, we introduce the SHOWMe dataset which consists of 96 videos, annotated with real and detailed hand-object 3D textured meshes. Following recent work, we consider a rigid hand-object scenario, in which the pose of the hand with respect to the object remains constant during the whole video sequence. This assumption allows us to register sub-millimetre-precise groundtruth 3D scans to the image sequences in SHOWMe. Although simpler, this hypothesis makes sense in terms of applications where the required accuracy and level of detail is important eg., object hand-over in human-robot collaboration, object scanning, or manipulation and contact point analysis. Importantly, the rigidity of the hand-object systems allows to tackle video-based 3D reconstruction of unknown hand-held objects using a 2-stage pipeline consisting of a rigid registration step followed by a multi-view reconstruction (MVR) part. We carefully evaluate a set of non-trivial baselines for these two stages and show that it is possible to achieve promising object-agnostic 3D hand-object reconstructions employing an SfM toolbox or a hand pose estimator to recover the rigid transforms and off-the-shelf MVR algorithms. However, these methods remain sensitive to the initial camera pose estimates which might be imprecise due to lack of textures on the objects or heavy occlusions of the hands, leaving room for improvements in the reconstruction. Code and dataset are available at https://europe.naverlabs.com/research/showme