Logos as a Well-Tempered Pre-train for Sign Language Recognition

TL;DR

This paper introduces Logos, a large and well-annotated Russian Sign Language dataset, demonstrating that pre-training on it enhances sign language recognition across languages and improves accuracy, especially for low-resource datasets.

Contribution

It presents the Logos dataset as the largest RSL dataset, and shows that pre-training on it serves as a universal encoder for various sign language recognition tasks, including cross-language transfer learning.

Findings

Pre-trained models on Logos outperform state-of-the-art on WLASL.

Joint training with multiple classification heads improves low-resource dataset accuracy.

Explicitly labeling visually similar signs enhances model quality.

Abstract

This paper examines two aspects of the isolated sign language recognition (ISLR) task. First, although a certain number of datasets is available, the data for individual sign languages is limited. It poses the challenge of cross-language ISLR model training, including transfer learning. Second, similar signs can have different semantic meanings. It leads to ambiguity in dataset labeling and raises the question of the best policy for annotating such signs. To address these issues, this study presents Logos, a novel Russian Sign Language (RSL) dataset, the most extensive available ISLR dataset by the number of signers, one of the most extensive datasets in size and vocabulary, and the largest RSL dataset. It is shown that a model, pre-trained on the Logos dataset can be used as a universal encoder for other language SLR tasks, including few-shot learning. We explore cross-language transfer learning approaches and find that joint training using multiple classification heads benefits accuracy for the target low-resource datasets the most. The key feature of the Logos dataset is explicitly annotated visually similar sign groups. We show that explicitly labeling visually similar signs improves trained model quality as a visual encoder for downstream tasks. Based on the proposed contributions, we outperform current state-of-the-art results for the WLASL dataset and get competitive results for the AUTSL dataset, with a single stream model processing solely RGB video. The source code, dataset, and pre-trained models are publicly available.