Snap, Segment, Deploy: A Visual Data and Detection Pipeline for Wearable Industrial Assistants

TL;DR

This paper presents a fully on-device, modular system for industrial assistance that combines lightweight perception, speech recognition, and retrieval techniques to support real-time, privacy-preserving training and operations in constrained environments.

Contribution

It introduces a novel on-device pipeline integrating detection, speech, and RAG for industrial support, with an automated data pipeline and a two-stage refinement strategy for robustness.

Findings

Improved robustness to domain shifts and visual corruptions.

Positive user feedback on guidance clarity and interaction quality.

Effective real-time support without cloud reliance.

Abstract

Industrial assembly tasks increasingly demand rapid adaptation to complex procedures and varied components, yet are often conducted in environments with limited computing, connectivity, and strict privacy requirements. These constraints make conventional cloud-based or fully autonomous solutions impractical for factory deployment. This paper introduces a mobile-device-based assistant system for industrial training and operational support, enabling real-time, semi-hands-free interaction through on-device perception and voice interfaces. The system integrates lightweight object detection, speech recognition, and Retrieval-Augmented Generation (RAG) into a modular on-device pipeline that operates entirely on-device, enabling intuitive support for part handling and procedure understanding without relying on manual supervision or cloud services. To enable scalable training, we adopt an automated data construction pipeline and introduce a two-stage refinement strategy to improve visual robustness under domain shift. Experiments on our generated dataset, i.e., Gear8, demonstrate improved robustness to domain shift and common visual corruptions. A structured user study further confirms its practical viability, with positive user feedback on the clarity of the guidance and the quality of the interaction. These results indicate that our framework offers a deployable solution for real-time, privacy-preserving smart assistance in industrial environments. We will release the Gear8 dataset and source code upon acceptance.