SceneComplete: Open-World 3D Scene Completion in Cluttered Real World Environments for Robot Manipulation

TL;DR

SceneComplete is a system that constructs detailed 3D models of cluttered environments from a single RGB-D image, enabling improved robot manipulation and grasping in complex scenes.

Contribution

It introduces a novel pipeline that combines multiple pretrained perception modules for accurate 3D scene reconstruction from limited input.

Findings

Achieves high accuracy in 3D scene reconstruction compared to ground-truth models

Enables robust grasp proposal generation in cluttered environments

Demonstrates effectiveness on a large benchmark dataset

Abstract

Careful robot manipulation in every-day cluttered environments requires an accurate understanding of the 3D scene, in order to grasp and place objects stably and reliably and to avoid colliding with other objects. In general, we must construct such a 3D interpretation of a complex scene based on limited input, such as a single RGB-D image. We describe SceneComplete, a system for constructing a complete, segmented, 3D model of a scene from a single view. SceneComplete is a novel pipeline for composing general-purpose pretrained perception modules (vision-language, segmentation, image-inpainting, image-to-3D, visual-descriptors and pose-estimation) to obtain highly accurate results. We demonstrate its accuracy and effectiveness with respect to ground-truth models in a large benchmark dataset and show that its accurate whole-object reconstruction enables robust grasp proposal generation, including for a dexterous hand. We release the code and additional results on our website.