Retrieval-Augmented Robots via Retrieve-Reason-Act

TL;DR

This paper introduces Retrieval-Augmented Robotics (RAR), enabling robots to actively retrieve and utilize external visual documentation for complex task execution, significantly improving performance in long-horizon assembly tasks.

Contribution

The paper formulates a novel Retrieve-Reason-Act paradigm for robots, integrating external visual manuals into the planning process for improved zero-shot task execution.

Findings

Grounded visual retrieval improves assembly success rates

Retrieval-based planning outperforms zero-shot baselines

Demonstrates effective external knowledge integration in robotics

Abstract

To achieve general-purpose utility, we argue that robots must evolve from passive executors into active Information Retrieval users. In strictly zero-shot settings where no prior demonstrations exist, robots face a critical information gap, such as the exact sequence required to assemble a complex furniture kit, that cannot be satisfied by internal parametric knowledge (common sense) or past internal memory. While recent robotic works attempt to use search before action, they primarily focus on retrieving past kinematic trajectories (analogous to searching internal memory) or text-based safety rules (searching for constraints). These approaches fail to address the core information need of active task construction: acquiring unseen procedural knowledge from external, unstructured documentation. In this paper, we define the paradigm as Retrieval-Augmented Robotics (RAR), empowering the robot with the information-seeking capability that bridges the gap between visual documentation and physical actuation. We formulate the task execution as an iterative Retrieve-Reason-Act loop: the robot or embodied agent actively retrieves relevant visual procedural manuals from an unstructured corpus, grounds the abstract 2D diagrams to 3D physical parts via cross-modal alignment, and synthesizes executable plans. We validate this paradigm on a challenging long-horizon assembly benchmark. Our experiments demonstrate that grounding robotic planning in retrieved visual documents significantly outperforms baselines relying on zero-shot reasoning or few-shot example retrieval. This work establishes the basis of RAR, extending the scope of Information Retrieval from answering user queries to driving embodied physical actions.