The American Sign Language Knowledge Graph: Infusing ASL Models with Linguistic Knowledge

TL;DR

This paper introduces the American Sign Language Knowledge Graph (ASLKG), a comprehensive resource built from linguistic sources, to enhance ASL understanding models with improved accuracy and explainability.

Contribution

The paper presents the ASLKG, a novel knowledge graph integrating linguistic knowledge, and demonstrates its effectiveness in training neuro-symbolic models for multiple ASL understanding tasks.

Findings

Achieved 91% accuracy on isolated sign recognition

Predicted semantic features of unseen signs with 14% accuracy

Classified YouTube-ASL video topics with 36% accuracy

Abstract

Language models for American Sign Language (ASL) could make language technologies substantially more accessible to those who sign. To train models on tasks such as isolated sign recognition (ISR) and ASL-to-English translation, datasets provide annotated video examples of ASL signs. To facilitate the generalizability and explainability of these models, we introduce the American Sign Language Knowledge Graph (ASLKG), compiled from twelve sources of expert linguistic knowledge. We use the ASLKG to train neuro-symbolic models for 3 ASL understanding tasks, achieving accuracies of 91% on ISR, 14% for predicting the semantic features of unseen signs, and 36% for classifying the topic of Youtube-ASL videos.