The Influence of Iconicity in Transfer Learning for Sign Language Recognition

TL;DR

This study investigates how the iconicity of signs affects transfer learning effectiveness in sign language recognition, comparing cross-linguistic transfer performance between different sign language pairs.

Contribution

It demonstrates the impact of sign iconicity on transfer learning success, highlighting differences across language pairs using a novel experimental setup.

Findings

7.02% improvement in Arabic sign recognition

1.07% improvement in Flemish sign recognition

Iconicity influences transfer learning effectiveness

Abstract

Most sign language recognition research relies on Transfer Learning (TL) from vision-based datasets such as ImageNet. Some extend this to alternatively available language datasets, often focusing on signs with cross-linguistic similarities. This body of work examines the necessity of these likenesses on effective knowledge transfer by comparing TL performance between iconic signs of two different sign language pairs: Chinese to Arabic and Greek to Flemish. Google Mediapipe was utilised as an input feature extractor, enabling spatial information of these signs to be processed with a Multilayer Perceptron architecture and the temporal information with a Gated Recurrent Unit. Experimental results showed a 7.02% improvement for Arabic and 1.07% for Flemish when conducting iconic TL from Chinese and Greek respectively.