MHB: Multimodal Handshape-aware Boundary Detection for Continuous Sign Language Recognition

TL;DR

This paper introduces a multimodal approach combining skeletal features and handshape information to improve boundary detection and recognition in continuous sign language videos, achieving significant accuracy gains.

Contribution

It presents a novel multimodal fusion framework that integrates 3D skeletal and handshape features for more robust boundary detection in continuous sign language recognition.

Findings

Significant improvement over previous methods on ASLLRP corpus

Effective use of 3D skeletal features for boundary detection

Successful integration of handshape classification into segmentation pipeline

Abstract

This paper employs a multimodal approach for continuous sign recognition by first using ML for detecting the start and end frames of signs in videos of American Sign Language (ASL) sentences, and then by recognizing the segmented signs. For improved robustness we use 3D skeletal features extracted from sign language videos to take into account the convergence of sign properties and their dynamics that tend to cluster at sign boundaries. Another focus of this paper is the incorporation of information from 3D handshape for boundary detection. To detect handshapes normally expected at the beginning and end of signs, we pretrain a handshape classifier for detection of 87 linguistically defined canonical handshape categories using a dataset that we created by integrating and normalizing several existing datasets. A multimodal fusion module is then used to unify the pretrained sign video segmentation framework and handshape classification models. Finally, the estimated boundaries are used for sign recognition, where the recognition model is trained on a large database containing both citation-form isolated signs and signs pre-segmented (based on manual annotations) from continuous signing-as such signs often differ a bit in certain respects. We evaluate our method on the ASLLRP corpus and demonstrate significant improvements over previous work.