Augmented Assembly: Object Recognition and Hand Tracking for Adaptive Assembly Instructions in Augmented Reality

TL;DR

This paper introduces an AR system that uses object recognition and hand tracking to assist users in assembly tasks by providing real-time guidance, detecting errors, and adapting instructions dynamically to enhance the assembly experience.

Contribution

It presents a novel AR-assisted assembly workflow integrating object recognition and hand tracking for real-time guidance and adaptive instructions in physical assembly tasks.

Findings

Real-time object detection and localization of components.

Dynamic AR overlay of instructions and error detection.

Supports flexible, non-linear assembly processes.

Abstract

Recent advances in augmented reality (AR) have enabled interactive systems that assist users in physical assembly tasks. In this paper, we present an AR-assisted assembly workflow that leverages object recognition and hand tracking to (1) identify custom components, (2) display step-by-step instructions, (3) detect assembly deviations, and (4) dynamically update the instructions based on users' hands-on interactions with physical parts. Using object recognition, the system detects and localizes components in real time to create a digital twin of the workspace. For each assembly step, it overlays bounding boxes in AR to indicate both the current position and the target placement of relevant components, while hand-tracking data verifies whether the user interacts with the correct part. Rather than enforcing a fixed sequence, the system highlights potential assembly errors and interprets user deviations as opportunities for iteration and creative exploration. A case study with LEGO blocks and custom 3D-printed components demonstrates how the system links digital instructions to physical assembly, eliminating the need for manual searching, sorting, or labeling of parts.