Improving Continuous Sign Language Recognition with Cross-Lingual Signs

TL;DR

This paper enhances continuous sign language recognition by leveraging cross-lingual signs from multiple sign languages, using a novel approach to identify and utilize shared signs as auxiliary data, leading to improved recognition accuracy.

Contribution

The work introduces a method to incorporate multilingual sign language data into CSLR by identifying cross-lingual signs and using them as auxiliary training data, which is a novel approach.

Findings

Achieves state-of-the-art results on Phoenix-2014 datasets.

Demonstrates the effectiveness of cross-lingual sign transfer.

Improves recognition accuracy by leveraging multilingual data.

Abstract

This work dedicates to continuous sign language recognition (CSLR), which is a weakly supervised task dealing with the recognition of continuous signs from videos, without any prior knowledge about the temporal boundaries between consecutive signs. Data scarcity heavily impedes the progress of CSLR. Existing approaches typically train CSLR models on a monolingual corpus, which is orders of magnitude smaller than that of speech recognition. In this work, we explore the feasibility of utilizing multilingual sign language corpora to facilitate monolingual CSLR. Our work is built upon the observation of cross-lingual signs, which originate from different sign languages but have similar visual signals (e.g., hand shape and motion). The underlying idea of our approach is to identify the cross-lingual signs in one sign language and properly leverage them as auxiliary training data to improve the recognition capability of another. To achieve the goal, we first build two sign language dictionaries containing isolated signs that appear in two datasets. Then we identify the sign-to-sign mappings between two sign languages via a well-optimized isolated sign language recognition model. At last, we train a CSLR model on the combination of the target data with original labels and the auxiliary data with mapped labels. Experimentally, our approach achieves state-of-the-art performance on two widely-used CSLR datasets: Phoenix-2014 and Phoenix-2014T.