Natural Language-Assisted Sign Language Recognition

TL;DR

This paper introduces NLA-SLR, a framework that leverages semantic information from glosses to improve sign language recognition, addressing visual ambiguities and enhancing model accuracy.

Contribution

It proposes language-aware label smoothing, inter-modality mixup, and a novel video-keypoint backbone to advance sign language recognition performance.

Findings

Achieves state-of-the-art results on MSASL, WLASL, and NMFs-CSL datasets.

Effectively distinguishes visually similar signs using semantic-aware techniques.

Enhances recognition accuracy by integrating multimodal features.

Abstract

Sign languages are visual languages which convey information by signers' handshape, facial expression, body movement, and so forth. Due to the inherent restriction of combinations of these visual ingredients, there exist a significant number of visually indistinguishable signs (VISigns) in sign languages, which limits the recognition capacity of vision neural networks. To mitigate the problem, we propose the Natural Language-Assisted Sign Language Recognition (NLA-SLR) framework, which exploits semantic information contained in glosses (sign labels). First, for VISigns with similar semantic meanings, we propose language-aware label smoothing by generating soft labels for each training sign whose smoothing weights are computed from the normalized semantic similarities among the glosses to ease training. Second, for VISigns with distinct semantic meanings, we present an inter-modality mixup technique which blends vision and gloss features to further maximize the separability of different signs under the supervision of blended labels. Besides, we also introduce a novel backbone, video-keypoint network, which not only models both RGB videos and human body keypoints but also derives knowledge from sign videos of different temporal receptive fields. Empirically, our method achieves state-of-the-art performance on three widely-adopted benchmarks: MSASL, WLASL, and NMFs-CSL. Codes are available at https://github.com/FangyunWei/SLRT.