Multitask training with unlabeled data for end-to-end sign language fingerspelling recognition

TL;DR

This paper presents a novel end-to-end model for American Sign Language fingerspelling recognition that leverages unlabeled video data through auto-encoder features, significantly improving accuracy without relying on frame-level labels.

Contribution

Introduces a joint auto-encoder and attention-based encoder-decoder model that utilizes unlabeled data for improved fingerspelling recognition.

Findings

Achieved 11.6% and 4.4% accuracy improvements over previous methods.

Effectively leverages unlabeled data to enhance feature learning.

Reduces dependency on frame-level labels and handcrafted features.

Abstract

We address the problem of automatic American Sign Language fingerspelling recognition from video. Prior work has largely relied on frame-level labels, hand-crafted features, or other constraints, and has been hampered by the scarcity of data for this task. We introduce a model for fingerspelling recognition that addresses these issues. The model consists of an auto-encoder-based feature extractor and an attention-based neural encoder-decoder, which are trained jointly. The model receives a sequence of image frames and outputs the fingerspelled word, without relying on any frame-level training labels or hand-crafted features. In addition, the auto-encoder subcomponent makes it possible to leverage unlabeled data to improve the feature learning. The model achieves 11.6% and 4.4% absolute letter accuracy improvement respectively in signer-independent and signer-adapted fingerspelling recognition over previous approaches that required frame-level training labels.