Optimizing Hand Region Detection in MediaPipe Holistic Full-Body Pose Estimation to Improve Accuracy and Avoid Downstream Errors

TL;DR

This paper improves hand region detection in MediaPipe Holistic by using a data-driven approach with enriched features, leading to more accurate ROI estimation and better sign language recognition performance.

Contribution

It introduces a novel data-driven method that enhances hand ROI prediction by incorporating additional keypoints and depth information.

Findings

Higher Intersection-over-Union scores for hand ROI estimation

Improved sign language recognition accuracy

Enhanced robustness to non-ideal hand orientations

Abstract

This paper addresses a critical flaw in MediaPipe Holistic's hand Region of Interest (ROI) prediction, which struggles with non-ideal hand orientations, affecting sign language recognition accuracy. We propose a data-driven approach to enhance ROI estimation, leveraging an enriched feature set including additional hand keypoints and the z-dimension. Our results demonstrate better estimates, with higher Intersection-over-Union compared to the current method. Our code and optimizations are available at https://github.com/sign-language-processing/mediapipe-hand-crop-fix.