Affordance RAG: Hierarchical Multimodal Retrieval with Affordance-Aware Embodied Memory for Mobile Manipulation

TL;DR

This paper introduces Affordance RAG, a hierarchical multimodal retrieval system that enables robots to understand and execute open-vocabulary manipulation tasks from natural language instructions, improving success rates in real-world indoor environments.

Contribution

The paper presents a novel zero-shot hierarchical multimodal retrieval framework incorporating affordance-aware embodied memory for mobile manipulation tasks.

Findings

Outperformed existing methods in retrieval accuracy for manipulation instructions.

Achieved 85% task success rate in real-world indoor robot experiments.

Demonstrated effective handling of free-form natural language commands.

Abstract

In this study, we address the problem of open-vocabulary mobile manipulation, where a robot is required to carry a wide range of objects to receptacles based on free-form natural language instructions. This task is challenging, as it involves understanding visual semantics and the affordance of manipulation actions. To tackle these challenges, we propose Affordance RAG, a zero-shot hierarchical multimodal retrieval framework that constructs Affordance-Aware Embodied Memory from pre-explored images. The model retrieves candidate targets based on regional and visual semantics and reranks them with affordance scores, allowing the robot to identify manipulation options that are likely to be executable in real-world environments. Our method outperformed existing approaches in retrieval performance for mobile manipulation instruction in large-scale indoor environments. Furthermore, in real-world experiments where the robot performed mobile manipulation in indoor environments based on free-form instructions, the proposed method achieved a task success rate of 85%, outperforming existing methods in both retrieval performance and overall task success.