Robustifying Long-term Human-Robot Collaboration through a Multimodal and Hierarchical Framework

TL;DR

This paper introduces a multimodal hierarchical framework for long-term human-robot collaboration, improving robustness, efficiency, and user satisfaction through integrated perception, plan prediction, and online adaptation.

Contribution

It presents a novel multimodal and hierarchical framework that enhances understanding and assistance in long-term HRC, with real-world deployment and extensive user studies.

Findings

Reduced task completion time by 15.9%

Achieved 91.8% success rate in tasks

User satisfaction score of 84%

Abstract

Long-term Human-Robot Collaboration (HRC) is crucial for enabling flexible manufacturing systems and integrating companion robots into daily human environments over extended periods. This paper identifies several key challenges for such collaborations, such as accurate recognition of human plan, robustness to disturbances, operational efficiency, adaptability to diverse user behaviors, and sustained human satisfaction. To address these challenges, we model the long-term HRC task through a hierarchical task graph and presents a novel multimodal and hierarchical framework to enable robots to better assist humans to advance on the task graph. In particular, the proposed multimodal framework integrates visual observations with speech commands to facilitate intuitive and flexible human-robot interactions. Additionally, our hierarchical designs for both human pose detection and plan prediction allow better understanding of human behaviors and significantly enhance system accuracy, robustness and flexibility. Moreover, an online adaptation mechanism enables real-time adjustment to diverse user behaviors. We deploy the proposed framework to KINOVA GEN3 robot and conduct extensive user studies on real-world long-term HRC assembly scenarios. Experimental results show that our approaches reduce task completion time by 15.9%, achieves an average task success rate of 91.8% and an overall user satisfaction score of 84% in long-term HRC tasks, showcasing its applicability in enhancing real-world long-term HRC.