Representing Signs as Signs: One-Shot ISLR to Facilitate Functional Sign Language Technologies

TL;DR

This paper introduces a one-shot learning approach for Isolated Sign Language Recognition that generalizes across languages and vocabularies, achieving state-of-the-art results and supporting scalable, adaptable sign language technology.

Contribution

It presents a novel one-shot ISLR method that generalizes across languages and vocabularies, enabling scalable sign language recognition.

Findings

Achieved 50.8% one-shot MRR on a large sign dictionary

State-of-the-art results in cross-language ISLR

Robust performance across different languages and datasets

Abstract

Isolated Sign Language Recognition (ISLR) is crucial for scalable sign language technology, yet language-specific approaches limit current models. To address this, we propose a one-shot learning approach that generalises across languages and evolving vocabularies. Our method involves pretraining a model to embed signs based on essential features and using a dense vector search for rapid, accurate recognition of unseen signs. We achieve state-of-the-art results, including 50.8% one-shot MRR on a large dictionary containing 10,235 unique signs from a different language than the training set. Our approach is robust across languages and support sets, offering a scalable, adaptable solution for ISLR. Co-created with the Deaf and Hard of Hearing (DHH) community, this method aligns with real-world needs, and advances scalable sign language recognition.