ShapeGrasp: Simultaneous Visuo-Haptic Shape Completion and Grasping for Improved Robot Manipulation

TL;DR

ShapeGrasp is a robotic system that combines visual and tactile feedback to iteratively complete object shapes and improve grasp success rates in real-world manipulation tasks.

Contribution

It introduces an iterative visuo-haptic shape completion and grasp planning pipeline that updates object shapes after each grasp attempt, a novel approach in real-world settings.

Findings

Achieved grasp success rates of 84% and 91% with different grippers.

Improved 3D shape reconstruction quality after each grasp.

First approach to update shape representations following real-world grasps.

Abstract

Humans grasp unfamiliar objects by combining an initial visual estimate with tactile and proprioceptive feedback during interaction. We present ShapeGrasp, a robotic implementation of this approach. The proposed method is an iterative grasp-and-complete pipeline that couples implicit surface visuo-haptic shape completion (creation of full 3D shape from partial information) with physics-based grasp planning. From a single RGB-D view, ShapeGrasp infers a complete shape (point cloud or triangular mesh), generates candidate grasps via rigid-body simulation, and executes the best feasible grasp. Each grasp attempt yields additional geometric constraints -- tactile surface contacts and space occupied by the gripper body -- which are fused to update the object shape. Failures trigger pose re-estimation and regrasping using the refined shape. We evaluate ShapeGrasp in the real world using two different robots and grippers. To the best of our knowledge, this is the first approach that updates shape representations following a real-world grasp. We achieved superior results over baselines for both grippers (grasp success rate of 84% with a three-finger gripper and 91% with a two-finger gripper), while improving the 3D shape reconstruction quality in all evaluation metrics used.