Self-supervised Optimization of Hand Pose Estimation using Anatomical Features and Iterative Learning

TL;DR

This paper introduces a self-supervised pipeline that adapts hand pose estimation models to specific scenarios with minimal human input, improving activity recognition in manual assembly tasks.

Contribution

It presents a novel self-supervised approach combining anatomical constraints and iterative retraining for robust hand pose estimation in complex scenarios.

Findings

Enhanced hand pose estimation accuracy in assembly scenarios

Improved activity recognition performance using adapted models

Effective minimal-interaction adaptation method

Abstract

Manual assembly workers face increasing complexity in their work. Human-centered assistance systems could help, but object recognition as an enabling technology hinders sophisticated human-centered design of these systems. At the same time, activity recognition based on hand poses suffers from poor pose estimation in complex usage scenarios, such as wearing gloves. This paper presents a self-supervised pipeline for adapting hand pose estimation to specific use cases with minimal human interaction. This enables cheap and robust hand posebased activity recognition. The pipeline consists of a general machine learning model for hand pose estimation trained on a generalized dataset, spatial and temporal filtering to account for anatomical constraints of the hand, and a retraining step to improve the model. Different parameter combinations are evaluated on a publicly available and annotated dataset. The best parameter and model combination is then applied to unlabelled videos from a manual assembly scenario. The effectiveness of the pipeline is demonstrated by training an activity recognition as a downstream task in the manual assembly scenario.