Data-Efficient American Sign Language Recognition via Few-Shot Prototypical Networks

TL;DR

This paper introduces a few-shot learning framework using prototypical networks and a spatiotemporal graph encoder for efficient American Sign Language recognition, especially effective with limited data and unseen signs.

Contribution

It presents a novel few-shot prototypical network approach with a spatiotemporal graph encoder for sign language recognition, improving generalization in data-scarce scenarios.

Findings

Achieves 43.75% Top-1 accuracy on WLASL dataset.

Outperforms standard classifiers by over 13% in scarce data conditions.

Demonstrates 30% accuracy on unseen signs without fine-tuning.

Abstract

Isolated Sign Language Recognition (ISLR) is critical for bridging the communication gap between the Deaf and Hard-of-Hearing (DHH) community and the hearing world. However, robust ISLR is fundamentally constrained by data scarcity and the long-tail distribution of sign vocabulary, where gathering sufficient examples for thousands of unique signs is prohibitively expensive. Standard classification approaches struggle under these conditions, often overfitting to frequent classes while failing to generalize to rare ones. To address this bottleneck, we propose a Few-Shot Prototypical Network framework adapted for a skeleton based encoder. Unlike traditional classifiers that learn fixed decision boundaries, our approach utilizes episodic training to learn a semantic metric space where signs are classified based on their proximity to dynamic class prototypes. We integrate a Spatiotemporal Graph Convolutional Network (ST-GCN) with a novel Multi-Scale Temporal Aggregation (MSTA) module to capture both rapid and fluid motion dynamics. Experimental results on the WLASL dataset demonstrate the superiority of this metric learning paradigm: our model achieves 43.75% Top-1 and 77.10% Top-5 accuracy on the test set. Crucially, this outperforms a standard classification baseline sharing the identical backbone architecture by over 13%, proving that the prototypical training strategy effectively outperforms in a data scarce situation where standard classification fails. Furthermore, the model exhibits strong zero-shot generalization, achieving nearly 30% accuracy on the unseen SignASL dataset without fine-tuning, offering a scalable pathway for recognizing extensive sign vocabularies with limited data.