3D Reconstruction of Objects in Hands without Real World 3D Supervision

TL;DR

This paper introduces a method to reconstruct 3D objects held in hands from a single image without relying on real-world 3D supervision, leveraging in-the-wild videos and synthetic shape data to improve generalization.

Contribution

It proposes modules that utilize indirect 3D cues from videos and shape collections to train models for 3D reconstruction without real-world 3D supervision.

Findings

11.6% relative improvement on MOW dataset

Effective use of in-the-wild video data

Leverages synthetic 3D shape collections

Abstract

Prior works for reconstructing hand-held objects from a single image train models on images paired with 3D shapes. Such data is challenging to gather in the real world at scale. Consequently, these approaches do not generalize well when presented with novel objects in in-the-wild settings. While 3D supervision is a major bottleneck, there is an abundance of a) in-the-wild raw video data showing hand-object interactions and b) synthetic 3D shape collections. In this paper, we propose modules to leverage 3D supervision from these sources to scale up the learning of models for reconstructing hand-held objects. Specifically, we extract multiview 2D mask supervision from videos and 3D shape priors from shape collections. We use these indirect 3D cues to train occupancy networks that predict the 3D shape of objects from a single RGB image. Our experiments in the challenging object generalization setting on in-the-wild MOW dataset show 11.6% relative improvement over models trained with 3D supervision on existing datasets.