3D Reconstruction of Objects in Hands without Real World 3D Supervision
Aditya Prakash, Matthew Chang, Matthew Jin, Ruisen Tu, Saurabh Gupta

TL;DR
This paper introduces a method to reconstruct 3D objects held in hands from a single image without relying on real-world 3D supervision, leveraging in-the-wild videos and synthetic shape data to improve generalization.
Contribution
It proposes modules that utilize indirect 3D cues from videos and shape collections to train models for 3D reconstruction without real-world 3D supervision.
Findings
11.6% relative improvement on MOW dataset
Effective use of in-the-wild video data
Leverages synthetic 3D shape collections
Abstract
Prior works for reconstructing hand-held objects from a single image train models on images paired with 3D shapes. Such data is challenging to gather in the real world at scale. Consequently, these approaches do not generalize well when presented with novel objects in in-the-wild settings. While 3D supervision is a major bottleneck, there is an abundance of a) in-the-wild raw video data showing hand-object interactions and b) synthetic 3D shape collections. In this paper, we propose modules to leverage 3D supervision from these sources to scale up the learning of models for reconstructing hand-held objects. Specifically, we extract multiview 2D mask supervision from videos and 3D shape priors from shape collections. We use these indirect 3D cues to train occupancy networks that predict the 3D shape of objects from a single RGB image. Our experiments in the challenging object…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsHuman Pose and Action Recognition · Hand Gesture Recognition Systems · Infrared Thermography in Medicine
