RAM: Retrieval-Based Affordance Transfer for Generalizable Zero-Shot Robotic Manipulation

TL;DR

RAM introduces a retrieval-based framework that enables zero-shot robotic manipulation by transferring affordances from diverse out-of-domain data, demonstrating superior generalization across objects, environments, and embodiments in simulation and real-world tasks.

Contribution

The paper presents RAM, a novel retrieve-and-transfer approach that leverages a comprehensive affordance memory for zero-shot, embodiment-agnostic robotic manipulation from out-of-domain data.

Findings

Outperforms existing methods in diverse tasks

Effective zero-shot transfer of affordances

Demonstrates potential in downstream applications

Abstract

This work proposes a retrieve-and-transfer framework for zero-shot robotic manipulation, dubbed RAM, featuring generalizability across various objects, environments, and embodiments. Unlike existing approaches that learn manipulation from expensive in-domain demonstrations, RAM capitalizes on a retrieval-based affordance transfer paradigm to acquire versatile manipulation capabilities from abundant out-of-domain data. First, RAM extracts unified affordance at scale from diverse sources of demonstrations including robotic data, human-object interaction (HOI) data, and custom data to construct a comprehensive affordance memory. Then given a language instruction, RAM hierarchically retrieves the most similar demonstration from the affordance memory and transfers such out-of-domain 2D affordance to in-domain 3D executable affordance in a zero-shot and embodiment-agnostic manner. Extensive simulation and real-world evaluations demonstrate that our RAM consistently outperforms existing works in diverse daily tasks. Additionally, RAM shows significant potential for downstream applications such as automatic and efficient data collection, one-shot visual imitation, and LLM/VLM-integrated long-horizon manipulation. For more details, please check our website at https://yxkryptonite.github.io/RAM/.