Explore More Guidance: A Task-aware Instruction Network for Sign Language Translation Enhanced with Data Augmentation

TL;DR

This paper introduces TIN-SLT, a task-aware instruction network for sign language translation that leverages a new instruction module and data augmentation to improve translation accuracy using pre-trained models.

Contribution

The work proposes a novel task-aware instruction network with an instruction module and feature fusion strategy, enhancing sign language translation performance by utilizing pre-trained language models.

Findings

Outperforms previous methods by 1.65 BLEU-4 on PHOENIX-2014-T

Outperforms previous methods by 1.42 BLEU-4 on ASLG-PC12

Effective multi-level data augmentation improves translation quality.

Abstract

Sign language recognition and translation first uses a recognition module to generate glosses from sign language videos and then employs a translation module to translate glosses into spoken sentences. Most existing works focus on the recognition step, while paying less attention to sign language translation. In this work, we propose a task-aware instruction network, namely TIN-SLT, for sign language translation, by introducing the instruction module and the learning-based feature fuse strategy into a Transformer network. In this way, the pre-trained model's language ability can be well explored and utilized to further boost the translation performance. Moreover, by exploring the representation space of sign language glosses and target spoken language, we propose a multi-level data augmentation scheme to adjust the data distribution of the training set. We conduct extensive experiments on two challenging benchmark datasets, PHOENIX-2014-T and ASLG-PC12, on which our method outperforms former best solutions by 1.65 and 1.42 in terms of BLEU-4. Our code is published at https://github.com/yongcaoplus/TIN-SLT.