A Multimodal Dataset for Enhancing Industrial Task Monitoring and Engagement Prediction

TL;DR

This paper introduces the MIAM dataset, a comprehensive multimodal collection of industrial task videos with detailed annotations, and proposes a multimodal network to improve engagement prediction in human-robot collaboration.

Contribution

The paper presents a novel multimodal dataset capturing real-world industrial workflows and a fusion-based network for enhanced engagement prediction.

Findings

Improved accuracy in engagement state recognition

Multimodal data fusion enhances task monitoring

Dataset facilitates evaluation of action localization and object interaction

Abstract

Detecting and interpreting operator actions, engagement, and object interactions in dynamic industrial workflows remains a significant challenge in human-robot collaboration research, especially within complex, real-world environments. Traditional unimodal methods often fall short of capturing the intricacies of these unstructured industrial settings. To address this gap, we present a novel Multimodal Industrial Activity Monitoring (MIAM) dataset that captures realistic assembly and disassembly tasks, facilitating the evaluation of key meta-tasks such as action localization, object interaction, and engagement prediction. The dataset comprises multi-view RGB, depth, and Inertial Measurement Unit (IMU) data collected from 22 sessions, amounting to 290 minutes of untrimmed video, annotated in detail for task performance and operator behavior. Its distinctiveness lies in the integration of multiple data modalities and its emphasis on real-world, untrimmed industrial workflows-key for advancing research in human-robot collaboration and operator monitoring. Additionally, we propose a multimodal network that fuses RGB frames, IMU data, and skeleton sequences to predict engagement levels during industrial tasks. Our approach improves the accuracy of recognizing engagement states, providing a robust solution for monitoring operator performance in dynamic industrial environments. The dataset and code can be accessed from https://github.com/navalkishoremehta95/MIAM/.