TFS Recognition: Investigating MPH]{Thai Finger Spelling Recognition: Investigating MediaPipe Hands Potentials

TL;DR

This paper evaluates MediaPipe Hands for Thai Finger Spelling recognition across different signing schemes, finding it effective for static and dynamic schemes but less so for point-on-hand signing due to occlusion and handedness issues.

Contribution

It demonstrates the potential and limitations of MediaPipe Hands in recognizing various Thai Finger Spelling schemes, highlighting specific challenges in point-on-hand recognition.

Findings

84.57% accuracy on static and dynamic single-hand schemes

23.66% accuracy on point-on-hand scheme with MPH

Conventional training outperforms MPH on point-on-hand scheme

Abstract

Thai Finger Spelling (TFS) sign recognition could benefit a community of hearing-difficulty people in bridging to a major hearing population. With a relatively large number of alphabets, TFS employs multiple signing schemes. Two schemes of more common signing -- static and dynamic single-hand signing, widely used in other sign languages -- have been addressed in several previous works. To complete the TFS sign recognition, the remaining two of quite distinct signing schemes -- static and dynamic point-on-hand signing -- need to be sufficiently addressed. With the advent of many off-the-shelf hand skeleton prediction models and that training a model to recognize a sign language from scratch is expensive, we explore an approach building upon recently launched MediaPipe Hands (MPH). MPH is a high-precision well-trained model for hand-keypoint detection. We have investigated MPH on three TFS schemes: static-single-hand (S1), simplified dynamic-single-hand (S2) and static-point-on-hand (P1) schemes. Our results show that MPH can satisfactorily address single-hand schemes with accuracy of 84.57% on both S1 and S2. However, our finding reveals a shortcoming of MPH in addressing a point-on-hand scheme, whose accuracy is 23.66% on P1 conferring to 69.19% obtained from conventional classification trained from scratch. This shortcoming has been investigated and attributed to self occlusion and handedness.