OWMM-Agent: Open World Mobile Manipulation With Multi-modal Agentic Data Synthesis
Junting Chen, Haotian Liang, Lingxiao Du, Weiyun Wang, Mengkang Hu, Yao Mu, Wenhai Wang, Jifeng Dai, Ping Luo, Wenqi Shao, Lin Shao

TL;DR
This paper introduces OWMM-Agent, a multi-modal foundation model for open-world mobile manipulation that integrates scene understanding, robot state tracking, and multi-modal action generation, achieving state-of-the-art results.
Contribution
The paper presents the first dedicated foundation model for mobile manipulators with comprehensive scene understanding and decision-making capabilities.
Findings
Achieves state-of-the-art performance in open-world manipulation tasks.
Demonstrates strong zero-shot generalization in real-world scenarios.
Introduces an agentic data synthesis pipeline for domain adaptation.
Abstract
The rapid progress of navigation, manipulation, and vision models has made mobile manipulators capable in many specialized tasks. However, the open-world mobile manipulation (OWMM) task remains a challenge due to the need for generalization to open-ended instructions and environments, as well as the systematic complexity to integrate high-level decision making with low-level robot control based on both global scene understanding and current agent state. To address this complexity, we propose a novel multi-modal agent architecture that maintains multi-view scene frames and agent states for decision-making and controls the robot by function calling. A second challenge is the hallucination from domain shift. To enhance the agent performance, we further introduce an agentic data synthesis pipeline for the OWMM task to adapt the VLM model to our task domain with instruction fine-tuning. We…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
