Exploring Attention Mechanisms in Integration of Multi-Modal Information for Sign Language Recognition and Translation

TL;DR

This paper introduces a lightweight cross-attention based plugin module for multi-modal sign language recognition and translation, improving accuracy while maintaining computational efficiency through a two-stage training process.

Contribution

The work proposes a novel, lightweight cross-attention plugin for multi-modal integration in sign language tasks, enabling end-to-end training and reducing computational complexity.

Findings

Reduced WER by 0.9 in recognition task

Increased BLEU-4 score by 0.8 in translation task

Efficient multi-modal feature merging with minimal overhead

Abstract

Understanding intricate and fast-paced movements of body parts is essential for the recognition and translation of sign language. The inclusion of additional information intended to identify and locate the moving body parts has been an interesting research topic recently. However, previous works on using multi-modal information raise concerns such as sub-optimal multi-modal feature merging method, or the model itself being too computationally heavy. In our work, we have addressed such issues and used a plugin module based on cross-attention to properly attend to each modality with another. Moreover, we utilized 2-stage training to remove the dependency of separate feature extractors for additional modalities in an end-to-end approach, which reduces the concern about computational complexity. Besides, our additional cross-attention plugin module is very lightweight which doesn't add significant computational overhead on top of the original baseline. We have evaluated the performance of our approaches on the RWTH-PHOENIX-2014 dataset for sign language recognition and the RWTH-PHOENIX-2014T dataset for the sign language translation task. Our approach reduced the WER by 0.9 on the recognition task and increased the BLEU-4 scores by 0.8 on the translation task.