Benchmarking Adaptive Intelligence and Computer Vision on Human-Robot Collaboration

TL;DR

This paper presents a benchmarking study on adaptive intelligence in human-robot collaboration, demonstrating improved intention recognition accuracy and self-labeling efficiency to combat concept drift in manufacturing environments.

Contribution

It introduces a novel self-labeling mechanism and a custom state detection algorithm to enhance intention recognition and adapt to environmental changes in HRC systems.

Findings

MViT2 with skeleton preprocessing achieves 83% accuracy.

SLB mechanism attains 91% labeling accuracy.

Model performance scales rapidly with fine-tuning on self-labeled data.

Abstract

Human-Robot Collaboration (HRC) is vital in Industry 4.0, using sensors, digital twins, collaborative robots (cobots), and intention-recognition models to have efficient manufacturing processes. However, Concept Drift is a significant challenge, where robots struggle to adapt to new environments. We address concept drift by integrating Adaptive Intelligence and self-labeling (SLB) to improve the resilience of intention-recognition in an HRC system. Our methodology begins with data collection using cameras and weight sensors, which is followed by annotation of intentions and state changes. Then we train various deep learning models with different preprocessing techniques for recognizing and predicting the intentions. Additionally, we developed a custom state detection algorithm for enhancing the accuracy of SLB, offering precise state-change definitions and timestamps to label intentions. Our results show that the MViT2 model with skeletal posture preprocessing achieves an accuracy of 83% on our data environment, compared to the 79% accuracy of MViT2 without skeleton posture extraction. Additionally, our SLB mechanism achieves a labeling accuracy of 91%, reducing a significant amount of time that would've been spent on manual annotation. Lastly, we observe swift scaling of model performance that combats concept drift by fine tuning on different increments of self-labeled data in a shifted domain that has key differences from the original training environment.. This study demonstrates the potential for rapid deployment of intelligent cobots in manufacturing through the steps shown in our methodology, paving a way for more adaptive and efficient HRC systems.