Master Thesis: Neural Sign Language Translation by Learning Tokenization

TL;DR

This thesis introduces a multitask learning approach for neural sign language translation that develops a generic sign-level tokenization layer, improving translation quality and efficiency without relying on gloss annotations.

Contribution

It proposes a novel sign-level tokenization method leveraging transfer learning and multitask learning, outperforming traditional gloss-level approaches in sign language translation.

Findings

Achieved 5 BLEU-4 point improvement in translation quality

Enhanced efficiency using 3D-CNNs for faster processing

Demonstrated advantages of sign-level over gloss-level tokenization

Abstract

In this thesis, we propose a multitask learning based method to improve Neural Sign Language Translation (NSLT) consisting of two parts, a tokenization layer and Neural Machine Translation (NMT). The tokenization part focuses on how Sign Language (SL) videos should be represented to be fed into the other part. It has not been studied elaborately whereas NMT research has attracted several researchers contributing enormous advancements. Up to now, there are two main input tokenization levels, namely frame-level and gloss-level tokenization. Glosses are world-like intermediate presentation and unique to SLs. Therefore, we aim to develop a generic sign-level tokenization layer so that it is applicable to other domains without further effort. We begin with investigating current tokenization approaches and explain their weaknesses with several experiments. To provide a solution, we adapt Transfer Learning, Multitask Learning and Unsupervised Domain Adaptation into this research to leverage additional supervision. We succeed in enabling knowledge transfer between SLs and improve translation quality by 5 points in BLEU-4 and 8 points in ROUGE scores. Secondly, we show the effects of body parts by extensive experiments in all the tokenization approaches. Apart from these, we adopt 3D-CNNs to improve efficiency in terms of time and space. Lastly, we discuss the advantages of sign-level tokenization over gloss-level tokenization. To sum up, our proposed method eliminates the need for gloss level annotation to obtain higher scores by providing additional supervision by utilizing weak supervision sources.