Affordance RAG: Hierarchical Multimodal Retrieval with Affordance-Aware Embodied Memory for Mobile Manipulation
Ryosuke Korekata, Quanting Xie, Yonatan Bisk, Komei Sugiura

TL;DR
This paper introduces Affordance RAG, a hierarchical multimodal retrieval system that enables robots to understand and execute open-vocabulary manipulation tasks from natural language instructions, improving success rates in real-world indoor environments.
Contribution
The paper presents a novel zero-shot hierarchical multimodal retrieval framework incorporating affordance-aware embodied memory for mobile manipulation tasks.
Findings
Outperformed existing methods in retrieval accuracy for manipulation instructions.
Achieved 85% task success rate in real-world indoor robot experiments.
Demonstrated effective handling of free-form natural language commands.
Abstract
In this study, we address the problem of open-vocabulary mobile manipulation, where a robot is required to carry a wide range of objects to receptacles based on free-form natural language instructions. This task is challenging, as it involves understanding visual semantics and the affordance of manipulation actions. To tackle these challenges, we propose Affordance RAG, a zero-shot hierarchical multimodal retrieval framework that constructs Affordance-Aware Embodied Memory from pre-explored images. The model retrieves candidate targets based on regional and visual semantics and reranks them with affordance scores, allowing the robot to identify manipulation options that are likely to be executable in real-world environments. Our method outperformed existing approaches in retrieval performance for mobile manipulation instruction in large-scale indoor environments. Furthermore, in…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsRobot Manipulation and Learning · Multimodal Machine Learning Applications · Social Robot Interaction and HRI
