SR3D: Unleashing Single-view 3D Reconstruction for Transparent and Specular Object Grasping

TL;DR

SR3D is a training-free framework that enables robotic grasping of transparent and specular objects from a single view by reconstructing accurate 3D models and localizing objects within scenes using view and keypoint matching.

Contribution

It introduces a novel single-view 3D reconstruction method that does not require training, specifically designed for transparent and specular objects in robotic grasping.

Findings

Effective 3D reconstruction in simulation and real-world scenarios

Improved grasp detection accuracy for challenging materials

Utilizes view and keypoint matching for precise object localization

Abstract

Recent advancements in 3D robotic manipulation have improved grasping of everyday objects, but transparent and specular materials remain challenging due to depth sensing limitations. While several 3D reconstruction and depth completion approaches address these challenges, they suffer from setup complexity or limited observation information utilization. To address this, leveraging the power of single view 3D object reconstruction approaches, we propose a training free framework SR3D that enables robotic grasping of transparent and specular objects from a single view observation. Specifically, given single view RGB and depth images, SR3D first uses the external visual models to generate 3D reconstructed object mesh based on RGB image. Then, the key idea is to determine the 3D object's pose and scale to accurately localize the reconstructed object back into its original depth corrupted 3D scene. Therefore, we propose view matching and keypoint matching mechanisms,which leverage both the 2D and 3D's inherent semantic and geometric information in the observation to determine the object's 3D state within the scene, thereby reconstructing an accurate 3D depth map for effective grasp detection. Experiments in both simulation and real world show the reconstruction effectiveness of SR3D.