Stimulating Imagination: Towards General-purpose "Something Something Placement"

TL;DR

This paper introduces SPORT, a method for general-purpose object placement that leverages large vision models and diffusion-based pose estimation to enable robots to rearrange objects based on vague instructions, with minimal training.

Contribution

SPORT combines vision reasoning and diffusion models for 3D object placement, reducing training needs and enabling zero-shot generalization to unseen objects.

Findings

Effective in generating 3D goal poses for unseen objects

Seamless transfer from simulation to real-world environments

No fine-tuning required for different object types

Abstract

General-purpose object placement is a fundamental capability of an intelligent generalist robot: being capable of rearranging objects following precise human instructions even in novel environments. This work is dedicated to achieving general-purpose object placement with ``something something'' instructions. Specifically, we break the entire process down into three parts, including object localization, goal imagination and robot control, and propose a method named SPORT. SPORT leverages a pre-trained large vision model for broad semantic reasoning about objects, and learns a diffusion-based pose estimator to ensure physically-realistic results in 3D space. Only object types (movable or reference) are communicated between these two parts, which brings two benefits. One is that we can fully leverage the powerful ability of open-set object recognition and localization since no specific fine-tuning is needed for the robotic scenario. Moreover, the diffusion-based estimator only need to ``imagine" the object poses after the placement, while no necessity for their semantic information. Thus the training burden is greatly reduced and no massive training is required. The training data for the goal pose estimation is collected in simulation and annotated by using GPT-4. Experimental results demonstrate the effectiveness of our approach. SPORT can not only generate promising 3D goal poses for unseen simulated objects, but also be seamlessly applied to real-world settings.