End-to-End Speech Translation of Arabic to English Broadcast News

TL;DR

This paper develops and compares pipeline and end-to-end deep neural network speech translation systems for Arabic to English broadcast news, demonstrating the potential of end-to-end models with transfer learning and data augmentation.

Contribution

It presents the first end-to-end Arabic-English broadcast news speech translation system and evaluates different training scenarios including transfer learning and data augmentation.

Findings

End-to-end models outperform pipeline systems in certain scenarios.

Transfer learning improves translation accuracy.

Data augmentation enhances model robustness.

Abstract

Speech translation (ST) is the task of directly translating acoustic speech signals in a source language into text in a foreign language. ST task has been addressed, for a long time, using a pipeline approach with two modules : first an Automatic Speech Recognition (ASR) in the source language followed by a text-to-text Machine translation (MT). In the past few years, we have seen a paradigm shift towards the end-to-end approaches using sequence-to-sequence deep neural network models. This paper presents our efforts towards the development of the first Broadcast News end-to-end Arabic to English speech translation system. Starting from independent ASR and MT LDC releases, we were able to identify about 92 hours of Arabic audio recordings for which the manual transcription was also translated into English at the segment level. These data was used to train and compare pipeline and end-to-end speech translation systems under multiple scenarios including transfer learning and data augmentation techniques.