Transfer Learning for Cross-dataset Isolated Sign Language Recognition in Under-Resourced Datasets

TL;DR

This paper introduces a transfer learning benchmark for under-resourced sign language datasets, demonstrating that specialized supervised methods can outperform simple finetuning in cross-dataset recognition tasks.

Contribution

It provides a new benchmark for cross-dataset transfer learning in sign language recognition and evaluates multiple approaches on under-resourced datasets.

Findings

Supervised transfer learning methods outperform finetuning.

Temporal graph convolution enhances recognition performance.

Benchmark dataset facilitates future research in under-resourced SLR.

Abstract

Sign language recognition (SLR) has recently achieved a breakthrough in performance thanks to deep neural networks trained on large annotated sign datasets. Of the many different sign languages, these annotated datasets are only available for a select few. Since acquiring gloss-level labels on sign language videos is difficult, learning by transferring knowledge from existing annotated sources is useful for recognition in under-resourced sign languages. This study provides a publicly available cross-dataset transfer learning benchmark from two existing public Turkish SLR datasets. We use a temporal graph convolution-based sign language recognition approach to evaluate five supervised transfer learning approaches and experiment with closed-set and partial-set cross-dataset transfer learning. Experiments demonstrate that improvement over finetuning based transfer learning is possible with specialized supervised transfer learning methods.