Incremental Language Understanding for Online Motion Planning of Robot Manipulators

TL;DR

This paper presents a novel incremental language understanding framework that enables robots to adapt their motion plans in real-time during human-robot interaction, improving flexibility and responsiveness.

Contribution

It introduces a reasoning-based incremental parser integrated with online motion planning, allowing continuous adaptation to evolving speech input and dynamic task constraints.

Findings

Enhanced ability to handle speech corrections and clarifications

Real-time adaptation of robot motion plans in dynamic scenarios

Improved fluidity in human-robot collaboration

Abstract

Human-robot interaction requires robots to process language incrementally, adapting their actions in real-time based on evolving speech input. Existing approaches to language-guided robot motion planning typically assume fully specified instructions, resulting in inefficient stop-and-replan behavior when corrections or clarifications occur. In this paper, we introduce a novel reasoning-based incremental parser which integrates an online motion planning algorithm within the cognitive architecture. Our approach enables continuous adaptation to dynamic linguistic input, allowing robots to update motion plans without restarting execution. The incremental parser maintains multiple candidate parses, leveraging reasoning mechanisms to resolve ambiguities and revise interpretations when needed. By combining symbolic reasoning with online motion planning, our system achieves greater flexibility in handling speech corrections and dynamically changing constraints. We evaluate our framework in real-world human-robot interaction scenarios, demonstrating online adaptions of goal poses, constraints, or task objectives. Our results highlight the advantages of integrating incremental language understanding with real-time motion planning for natural and fluid human-robot collaboration. The experiments are demonstrated in the accompanying video at www.acin.tuwien.ac.at/42d5.