State Space Models are Effective Sign Language Learners: Exploiting Phonological Compositionality for Vocabulary-Scale Recognition

TL;DR

This paper introduces PHONSSM, a sign language recognition model that leverages phonological compositionality and graph attention to improve vocabulary-scale recognition, especially in few-shot and zero-shot scenarios.

Contribution

The paper presents PHONSSM, a novel architecture that enforces phonological decomposition for sign language recognition, achieving state-of-the-art results on large datasets using skeleton data.

Findings

PHONSSM achieves 72.1% accuracy on WLASL2000, outperforming previous skeleton-based methods.

The model shows a 225% improvement in few-shot recognition.

PHONSSM successfully transfers zero-shot to unseen signs, surpassing RGB-based baselines.

Abstract

Sign language recognition suffers from catastrophic scaling failure: models achieving high accuracy on small vocabularies collapse at realistic sizes. Existing architectures treat signs as atomic visual patterns, learning flat representations that cannot exploit the compositional structure of sign languages-systematically organized from discrete phonological parameters (handshape, location, movement, orientation) reused across the vocabulary. We introduce PHONSSM, enforcing phonological decomposition through anatomically-grounded graph attention, explicit factorization into orthogonal subspaces, and prototypical classification enabling few-shot transfer. Using skeleton data alone on the largest ASL dataset ever assembled (5,565 signs), PHONSSM achieves 72.1% on WLASL2000 (+18.4pp over skeleton SOTA), surpassing most RGB methods without video input. Gains are most dramatic in the few-shot regime (+225% relative), and the model transfers zero-shot to ASL Citizen, exceeding supervised RGB baselines. The vocabulary scaling bottleneck is fundamentally a representation learning problem, solvable through compositional inductive biases mirroring linguistic structure.