Optimizing Multitask Industrial Processes with Predictive Action Guidance

TL;DR

This paper presents a multimodal transformer-based system for egocentric activity anticipation in industrial assembly, providing proactive guidance and deviation detection to improve productivity and compliance.

Contribution

It introduces the MMTFRU network with multimodal fusion and integrated guidance strategies, advancing predictive accuracy and anomaly detection in complex industrial tasks.

Findings

Effective in predicting next actions in industrial assembly

Improves operator guidance and deviation detection

Validated on Meccano and EPIC-Kitchens-55 datasets

Abstract

Monitoring complex assembly processes is critical for maintaining productivity and ensuring compliance with assembly standards. However, variability in human actions and subjective task preferences complicate accurate task anticipation and guidance. To address these challenges, we introduce the Multi-Modal Transformer Fusion and Recurrent Units (MMTFRU) Network for egocentric activity anticipation, utilizing multimodal fusion to improve prediction accuracy. Integrated with the Operator Action Monitoring Unit (OAMU), the system provides proactive operator guidance, preventing deviations in the assembly process. OAMU employs two strategies: (1) Top-5 MMTF-RU predictions, combined with a reference graph and an action dictionary, for next-step recommendations; and (2) Top-1 MMTF-RU predictions, integrated with a reference graph, for detecting sequence deviations and predicting anomaly scores via an entropy-informed confidence mechanism. We also introduce Time-Weighted Sequence Accuracy (TWSA) to evaluate operator efficiency and ensure timely task completion. Our approach is validated on the industrial Meccano dataset and the largescale EPIC-Kitchens-55 dataset, demonstrating its effectiveness in dynamic environments.