Reconstructing Hand-Held Objects from Monocular Video

TL;DR

This paper introduces a monocular video-based method for reconstructing hand-held objects that leverages hand motion as multiple viewpoints, using implicit neural representations without requiring prior object models.

Contribution

It proposes a novel approach that reconstructs detailed object geometry from monocular video without learned priors, utilizing hand motion and implicit neural representations.

Findings

Achieves more accurate and detailed object reconstructions.

Validates approach on a new dataset with 3D ground truth.

Handles imprecise hand pose estimation and small objects effectively.

Abstract

This paper presents an approach that reconstructs a hand-held object from a monocular video. In contrast to many recent methods that directly predict object geometry by a trained network, the proposed approach does not require any learned prior about the object and is able to recover more accurate and detailed object geometry. The key idea is that the hand motion naturally provides multiple views of the object and the motion can be reliably estimated by a hand pose tracker. Then, the object geometry can be recovered by solving a multi-view reconstruction problem. We devise an implicit neural representation-based method to solve the reconstruction problem and address the issues of imprecise hand pose estimation, relative hand-object motion, and insufficient geometry optimization for small objects. We also provide a newly collected dataset with 3D ground truth to validate the proposed approach.