Towards Real-time Adaptation of Embodied Agent in Human-Robot Collaboration

TL;DR

This paper introduces MonTA, a hierarchical framework enabling real-time, low-latency adaptation in human-robot collaboration, validated through a new benchmark and user studies.

Contribution

The paper presents MonTA, a novel hierarchical framework with high-frequency monitoring and low-frequency adaptation modules, improving real-time responsiveness in embodied agents.

Findings

MonTA outperforms baseline agents on the Overcooked-AI benchmark.

User studies confirm the reasonableness and fluency of MonTA's adaptation plans.

MonTA achieves low latency at 7 Hz for monitoring and effective adaptation.

Abstract

Large Language Models (LLMs) have opened transformative possibilities for human-robot collaboration. However, enabling real-time collaboration requires both low latency and robust reasoning, and most LLMs suffer from high latency. To address this gap, we first propose a fine-grained benchmark that explicitly assesses agents' proactive adaptability and temporal responsiveness in the Overcooked-AI environment. Based on evaluation results, we propose MonTA (Monitor-then-Adapt), a hierarchical framework inspired by cognitive science research. MonTA contains three key modules: a lightweight Monitor that operates at high frequency (7 Hz) to detect adaptation needs, and two proficient Adapters for subtask and path adaptation reasoning that provide instructions to humans at a lower frequency. Our results demonstrate that MonTA significantly outperforms baseline agents on our proposed benchmark, achieving superior performance across layouts with varying teaming fluency. User studies confirm the high reasonableness of adaptation plans and consistent language instructions provided by our framework to humans.