Challenges of the Creation of a Dataset for Vision Based Human Hand Action Recognition in Industrial Assembly

TL;DR

This paper introduces the Industrial Hand Action Dataset V1, a comprehensive industrial assembly dataset with diverse annotations, and demonstrates its effectiveness by training a Gated Transformer Network achieving 86.25% accuracy.

Contribution

The creation of a large, annotated industrial hand action dataset with unique features like occlusions and interactions, and its use to train a state-of-the-art transformer model.

Findings

Dataset contains over 2 million images with diverse industrial hand actions.

Gated Transformer Network achieved 86.25% accuracy on the dataset.

Dataset meets technical and legal requirements for industrial applications.

Abstract

This work presents the Industrial Hand Action Dataset V1, an industrial assembly dataset consisting of 12 classes with 459,180 images in the basic version and 2,295,900 images after spatial augmentation. Compared to other freely available datasets tested, it has an above-average duration and, in addition, meets the technical and legal requirements for industrial assembly lines. Furthermore, the dataset contains occlusions, hand-object interaction, and various fine-grained human hand actions for industrial assembly tasks that were not found in combination in examined datasets. The recorded ground truth assembly classes were selected after extensive observation of real-world use cases. A Gated Transformer Network, a state-of-the-art model from the transformer domain was adapted, and proved with a test accuracy of 86.25% before hyperparameter tuning by 18,269,959 trainable parameters, that it is possible to train sequential deep learning models with this dataset.