Integrative Object and Pose to Task Detection for an Augmented-Reality-based Human Assistance System using Neural Networks

TL;DR

This paper presents an AR-based human assistance system that integrates neural network-driven object and pose detection to improve task efficiency and user acceptance in industrial settings.

Contribution

It introduces a novel integration of deep neural networks with AR for real-time object and pose detection to assist workers in complex manual tasks.

Findings

Significant reduction in task completion time for untrained workers

Decrease in error rates during task execution

Positive user acceptance and learning curve evidence

Abstract

As a result of an increasingly automatized and digitized industry, processes are becoming more complex. Augmented Reality has shown considerable potential in assisting workers with complex tasks by enhancing user understanding and experience with spatial information. However, the acceptance and integration of AR into industrial processes is still limited due to the lack of established methods and tedious integration efforts. Meanwhile, deep neural networks have achieved remarkable results in computer vision tasks and bear great prospects to enrich Augmented Reality applications . In this paper, we propose an Augmented-Reality-based human assistance system to assist workers in complex manual tasks where we incorporate deep neural networks for computer vision tasks. More specifically, we combine Augmented Reality with object and action detectors to make workflows more intuitive and flexible. To evaluate our system in terms of user acceptance and efficiency, we conducted several user studies. We found a significant reduction in time to task completion in untrained workers and a decrease in error rate. Furthermore, we investigated the users learning curve with our assistance system.