A Classification Model Utilizing Facial Landmark Tracking to Determine Sentence Types for American Sign Language Recognition

TL;DR

This paper presents a computationally efficient facial landmark-based classification model that improves American Sign Language sentence type recognition by distinguishing statements from assertions with 86.5% accuracy.

Contribution

It introduces a novel facial landmark tracking approach combined with PCA and Random Forests for classifying ASL sentence types, enhancing existing sign language recognition systems.

Findings

Achieved 86.5% accuracy in classifying sentence types

Utilized facial landmarks and PCA for feature extraction

Demonstrated the effectiveness of facial cues in ASL recognition

Abstract

The deaf and hard of hearing community relies on American Sign Language (ASL) as their primary mode of communication, but communication with others who do not know ASL can be difficult, especially during emergencies where no interpreter is available. As an effort to alleviate this problem, research in computer vision based real time ASL interpreting models is ongoing. However, most of these models are hand shape (gesture) based and lack the integration of facial cues, which are crucial in ASL to convey tone and distinguish sentence types. Thus, the integration of facial cues in computer vision based ASL interpreting models has the potential to improve performance and reliability. In this paper, we introduce a simple, computationally efficient facial expression based classification model that can be used to improve ASL interpreting models. This model utilizes the relative angles of facial landmarks with principal component analysis and a Random Forest Classification tree model to classify frames taken from videos of ASL users signing a complete sentence. The model classifies the frames as statements or assertions. The model was able to achieve an accuracy of 86.5%.