Mixed-Initiative Dialog for Human-Robot Collaborative Manipulation

TL;DR

This paper introduces MICoBot, a mixed-initiative dialog system for human-robot collaboration that dynamically allocates tasks and communicates via natural language, significantly improving success and user experience.

Contribution

The paper presents MICoBot, a novel multi-level decision system enabling flexible, natural language-based human-robot collaboration with adaptive task allocation and communication strategies.

Findings

MICoBot outperforms baseline models in task success.

Participants reported higher user experience with MICoBot.

The system effectively adapts to diverse human partners.

Abstract

Effective robotic systems for long-horizon human-robot collaboration must adapt to a wide range of human partners, whose physical behavior, willingness to assist, and understanding of the robot's capabilities may change over time. This demands a tightly coupled communication loop that grants both agents the flexibility to propose, accept, or decline requests as they coordinate toward completing the task effectively. We apply a Mixed-Initiative dialog paradigm to Collaborative human-roBot teaming and propose MICoBot, a system that handles the common scenario where both agents, using natural language, take initiative in formulating, accepting, or rejecting proposals on who can best complete different steps of a task. To handle diverse, task-directed dialog, and find successful collaborative strategies that minimize human effort, MICoBot makes decisions at three levels: (1) a meta-planner considers human dialog to formulate and code a high-level collaboration strategy, (2) a planner optimally allocates the remaining steps to either agent based on the robot's capabilities (measured by a simulation-pretrained affordance model) and the human's estimated availability to help, and (3) an action executor decides the low-level actions to perform or words to say to the human. In physical robot trials with 18 unique human participants, MICoBot significantly improves task success and user experience over a pure LLM baseline and standard agent allocation models. See additional videos and materials at https://robin-lab.cs.utexas.edu/MicoBot/.