Object Reconstruction under Occlusion with Generative Priors and Contact-induced Constraints

TL;DR

This paper presents a method for object reconstruction under occlusion by combining generative shape priors with contact-based constraints, improving accuracy in partial observations for robotic manipulation.

Contribution

It introduces a contact-guided 3D generation approach that integrates generative priors and contact information to enhance object reconstruction under occlusion.

Findings

Improved reconstruction accuracy over pure generative models.

Effective use of contact information from videos and interactions.

Validated on synthetic and real-world datasets.

Abstract

Object geometry is key information for robot manipulation. Yet, object reconstruction is a challenging task because cameras only capture partial observations of objects, especially when occlusion occurs. In this paper, we leverage two extra sources of information to reduce the ambiguity of vision signals. First, generative models learn priors of the shapes of commonly seen objects, allowing us to make reasonable guesses of the unseen part of geometry. Second, contact information, which can be obtained from videos and physical interactions, provides sparse constraints on the boundary of the geometry. We combine the two sources of information through contact-guided 3D generation. The guidance formulation is inspired by drag-based editing in generative models. Experiments on synthetic and real-world data show that our approach improves the reconstruction compared to pure 3D generation and contact-based optimization.