HandyLabel: Towards Post-Processing to Real-Time Annotation Using Skeleton Based Hand Gesture Recognition

TL;DR

HandyLabel is a real-time hand gesture annotation tool that improves data labeling efficiency by leveraging skeleton-based recognition and user customization, validated through model evaluation and user study.

Contribution

This work introduces HandyLabel, a novel real-time annotation system using skeleton-based hand gesture recognition with customizable mappings and validation on open datasets.

Findings

ResNet50 with skeleton preprocessing achieves 0.923 F1-score.

88.9% of users preferred HandyLabel over traditional tools.

HandyLabel enables efficient, real-time gesture annotation with high user satisfaction.

Abstract

The success of machine learning is deeply linked to the availability of high-quality training data, yet retrieving and manually labeling new data remains a time-consuming and error-prone process. Traditional annotation tools, such as Label Studio, often require post-processing, where users label data after it has been recorded. Post-processing is highly time-consuming and labor-intensive, especially with large datasets, and may lead to erroneous annotations due to the difficulty of subjects' memory tasks when labeling cognitive activities such as emotions or comprehension levels. In this work, we introduce HandyLabel, a real-time annotation tool that leverages hand gesture recognition to map hand signs for labeling. The application enables users to customize gesture mappings through a web-based interface, allowing for real-time annotations. To ensure the performance of HandyLabel, we evaluate several hand gesture recognition models on an open-source hand sign (HaGRID) dataset, with and without skeleton-based preprocessing. We discovered that ResNet50 with preprocessed skeleton-based images performs an F1-score of 0.923. To validate the usability of HandyLabel, a user study was conducted with 46 participants. The results suggest that 88.9% of participants preferred HandyLabel over traditional annotation tools.