Signer-independent Fingerspelling Recognition with Deep Neural Network Adaptation

TL;DR

This paper explores signer-independent fingerspelling recognition in American Sign Language using deep neural network adaptation, significantly improving accuracy with minimal adaptation data.

Contribution

It introduces DNN adaptation techniques for signer-independent fingerspelling recognition, reducing the performance gap with signer-dependent systems.

Findings

Achieved up to 82.7% letter accuracy with frame-level adaptation.

Achieved up to 69.7% letter accuracy with only word labels.

Demonstrated effectiveness of DNN adaptation with limited data.

Abstract

We study the problem of recognition of fingerspelled letter sequences in American Sign Language in a signer-independent setting. Fingerspelled sequences are both challenging and important to recognize, as they are used for many content words such as proper nouns and technical terms. Previous work has shown that it is possible to achieve almost 90% accuracies on fingerspelling recognition in a signer-dependent setting. However, the more realistic signer-independent setting presents challenges due to significant variations among signers, coupled with the dearth of available training data. We investigate this problem with approaches inspired by automatic speech recognition. We start with the best-performing approaches from prior work, based on tandem models and segmental conditional random fields (SCRFs), with features based on deep neural network (DNN) classifiers of letters and phonological features. Using DNN adaptation, we find that it is possible to bridge a large part of the gap between signer-dependent and signer-independent performance. Using only about 115 transcribed words for adaptation from the target signer, we obtain letter accuracies of up to 82.7% with frame-level adaptation labels and 69.7% with only word labels.