BINDER: Instantly Adaptive Mobile Manipulation with Open-Vocabulary Commands

TL;DR

BINDER is a dual-process framework that enables robots to adapt instantly to dynamic environments by combining strategic planning with continuous environment monitoring using multimodal large language models.

Contribution

It introduces a novel dual process approach that decouples planning from real-time monitoring, improving robustness and efficiency in open-vocabulary mobile manipulation.

Findings

BINDER outperforms state-of-the-art methods in success rate and efficiency.

It effectively handles dynamic object placement in real-world environments.

The framework demonstrates robust adaptation in three different real-world settings.

Abstract

Open-vocabulary mobile manipulation (OVMM) requires robots to follow language instructions, navigate, and manipulate while updating their world representation under dynamic environmental changes. However, most prior approaches update their world representation only at discrete update points such as navigation targets, waypoints, or the end of an action step, leaving robots blind between updates and causing cascading failures: overlooked objects, late error detection, and delayed replanning. To address this limitation, we propose BINDER (Bridging INstant and DEliberative Reasoning), a dual process framework that decouples strategic planning from continuous environment monitoring. Specifically, BINDER integrates a Deliberative Response Module (DRM, a multimodal LLM for task planning) with an Instant Response Module (IRM, a VideoLLM for continuous monitoring). The two modules play complementary roles: the DRM performs strategic planning with structured 3D scene updates and guides what the IRM attends to, while the IRM analyzes video streams to update memory, correct ongoing actions, and trigger replanning when necessary. Through this bidirectional coordination, the modules address the trade off between maintaining awareness and avoiding costly updates, enabling robust adaptation under dynamic conditions. Evaluated in three real world environments with dynamic object placement, BINDER achieves substantially higher success and efficiency than SoTA baselines, demonstrating its effectiveness for real world deployment.