Open-vocabulary Mobile Manipulation in Unseen Dynamic Environments with 3D Semantic Maps

TL;DR

This paper presents a novel framework enabling mobile robots to perform open-vocabulary manipulation in unseen dynamic environments by integrating 3D semantic mapping, visual-language models, and large language models for natural language understanding and planning.

Contribution

It introduces a new framework combining 3D semantic maps, visual-language models, and large language models for zero-shot manipulation and natural language understanding in dynamic environments.

Findings

Achieved over 80% success rate in real-world experiments.

Demonstrated effective zero-shot manipulation and natural language processing.

Improved navigation and task success metrics significantly over baseline methods.

Abstract

Open-Vocabulary Mobile Manipulation (OVMM) is a crucial capability for autonomous robots, especially when faced with the challenges posed by unknown and dynamic environments. This task requires robots to explore and build a semantic understanding of their surroundings, generate feasible plans to achieve manipulation goals, adapt to environmental changes, and comprehend natural language instructions from humans. To address these challenges, we propose a novel framework that leverages the zero-shot detection and grounded recognition capabilities of pretraining visual-language models (VLMs) combined with dense 3D entity reconstruction to build 3D semantic maps. Additionally, we utilize large language models (LLMs) for spatial region abstraction and online planning, incorporating human instructions and spatial semantic context. We have built a 10-DoF mobile manipulation robotic platform JSR-1 and demonstrated in real-world robot experiments that our proposed framework can effectively capture spatial semantics and process natural language user instructions for zero-shot OVMM tasks under dynamic environment settings, with an overall navigation and task success rate of 80.95% and 73.33% over 105 episodes, and better SFT and SPL by 157.18% and 19.53% respectively compared to the baseline. Furthermore, the framework is capable of replanning towards the next most probable candidate location based on the spatial semantic context derived from the 3D semantic map when initial plans fail, keeping an average success rate of 76.67%.