Grasp, See, and Place: Efficient Unknown Object Rearrangement with Policy Structure Prior

TL;DR

This paper introduces GSP, a dual-loop system for unknown object rearrangement that leverages a decoupled perception structure and foundation models to improve task success amidst perception noise.

Contribution

It presents a novel decoupled structure for perception in rearrangement tasks and a dual-loop system utilizing foundation models for improved performance.

Findings

Higher completion rates in rearrangement tasks

Fewer steps required for successful rearrangement

Robustness to perception noise demonstrated

Abstract

We focus on the task of unknown object rearrangement, where a robot is supposed to re-configure the objects into a desired goal configuration specified by an RGB-D image. Recent works explore unknown object rearrangement systems by incorporating learning-based perception modules. However, they are sensitive to perception error, and pay less attention to task-level performance. In this paper, we aim to develop an effective system for unknown object rearrangement amidst perception noise. We theoretically reveal that the noisy perception impacts grasp and place in a decoupled way, and show such a decoupled structure is valuable to improve task optimality. We propose GSP, a dual-loop system with the decoupled structure as prior. For the inner loop, we learn a see policy for self-confident in-hand object matching. For the outer loop, we learn a grasp policy aware of object matching and grasp capability guided by task-level rewards. We leverage the foundation model CLIP for object matching, policy learning and self-termination. A series of experiments indicate that GSP can conduct unknown object rearrangement with higher completion rates and fewer steps.