Fingerspelling Detection in American Sign Language

TL;DR

This paper introduces a new approach for detecting fingerspelling in raw American Sign Language videos, addressing the challenge of untrimmed video analysis and advancing towards real-world recognition systems.

Contribution

It presents a benchmark, evaluation metrics, and a novel multi-task learning model that integrates pose estimation and recognition for improved fingerspelling detection.

Findings

The proposed model outperforms alternatives across all metrics.

Established a new state-of-the-art on the benchmark.

Demonstrated effectiveness in untrimmed video detection scenarios.

Abstract

Fingerspelling, in which words are signed letter by letter, is an important component of American Sign Language. Most previous work on automatic fingerspelling recognition has assumed that the boundaries of fingerspelling regions in signing videos are known beforehand. In this paper, we consider the task of fingerspelling detection in raw, untrimmed sign language videos. This is an important step towards building real-world fingerspelling recognition systems. We propose a benchmark and a suite of evaluation metrics, some of which reflect the effect of detection on the downstream fingerspelling recognition task. In addition, we propose a new model that learns to detect fingerspelling via multi-task training, incorporating pose estimation and fingerspelling recognition (transcription) along with detection, and compare this model to several alternatives. The model outperforms all alternative approaches across all metrics, establishing a state of the art on the benchmark.