The Sem-Lex Benchmark: Modeling ASL Signs and Their Phonemes

TL;DR

The Sem-Lex Benchmark is a large, annotated ASL dataset that improves sign recognition accuracy by incorporating phonological features, advancing research in sign language technology.

Contribution

Introduces the Sem-Lex Benchmark, the largest ASL dataset with phonological annotations, and demonstrates its effectiveness for improving isolated sign recognition.

Findings

Phonological features are recognizable with 85% accuracy.

Using phonological features improves few-shot ISR accuracy by 6%.

Overall ISR accuracy improves by 2% when using phonological features.

Abstract

Sign language recognition and translation technologies have the potential to increase access and inclusion of deaf signing communities, but research progress is bottlenecked by a lack of representative data. We introduce a new resource for American Sign Language (ASL) modeling, the Sem-Lex Benchmark. The Benchmark is the current largest of its kind, consisting of over 84k videos of isolated sign productions from deaf ASL signers who gave informed consent and received compensation. Human experts aligned these videos with other sign language resources including ASL-LEX, SignBank, and ASL Citizen, enabling useful expansions for sign and phonological feature recognition. We present a suite of experiments which make use of the linguistic information in ASL-LEX, evaluating the practicality and fairness of the Sem-Lex Benchmark for isolated sign recognition (ISR). We use an SL-GCN model to show that the phonological features are recognizable with 85% accuracy, and that they are effective as an auxiliary target to ISR. Learning to recognize phonological features alongside gloss results in a 6% improvement for few-shot ISR accuracy and a 2% improvement for ISR accuracy overall. Instructions for downloading the data can be found at https://github.com/leekezar/SemLex.