Hand Guided High Resolution Feature Enhancement for Fine-Grained Atomic Action Segmentation within Complex Human Assemblies

TL;DR

This paper introduces a high-resolution, hand-guided feature enhancement model for fine-grained atomic action segmentation in complex human assemblies, enabling real-time, accurate classification in manufacturing settings.

Contribution

It proposes a novel hand location guided high-resolution feature enhancement approach and a method for real-time segmentation using offline-trained models with surround sampling and label cleaning.

Findings

High-resolution hand features improve action classification accuracy.

The model surpasses similar encoder/decoder methods in real-time segmentation.

Effective for fine-grained atomic actions in manufacturing environments.

Abstract

Due to the rapid temporal and fine-grained nature of complex human assembly atomic actions, traditional action segmentation approaches requiring the spatial (and often temporal) down sampling of video frames often loose vital fine-grained spatial and temporal information required for accurate classification within the manufacturing domain. In order to fully utilise higher resolution video data (often collected within the manufacturing domain) and facilitate real time accurate action segmentation - required for human robot collaboration - we present a novel hand location guided high resolution feature enhanced model. We also propose a simple yet effective method of deploying offline trained action recognition models for real time action segmentation on temporally short fine-grained actions, through the use of surround sampling while training and temporally aware label cleaning at inference. We evaluate our model on a novel action segmentation dataset containing 24 (+background) atomic actions from video data of a real world robotics assembly production line. Showing both high resolution hand features as well as traditional frame wide features improve fine-grained atomic action classification, and that though temporally aware label clearing our model is capable of surpassing similar encoder/decoder methods, while allowing for real time classification.