Speech-to-Speech Translation For A Real-world Unwritten Language

TL;DR

This paper develops an end-to-end speech-to-speech translation system for unwritten languages, using English-Taiwanese Hokkien as a case study, and introduces new data collection, modeling techniques, and a benchmark dataset.

Contribution

It presents novel methods for data collection, weak supervision, and leveraging related languages, along with a benchmark dataset for unwritten language translation.

Findings

Effective use of pseudo-labeling for weakly supervised data

Leveraging related language (Mandarin) improves translation quality

Release of a new benchmark dataset for unwritten language S2ST

Abstract

We study speech-to-speech translation (S2ST) that translates speech from one language into another language and focuses on building systems to support languages without standard text writing systems. We use English-Taiwanese Hokkien as a case study, and present an end-to-end solution from training data collection, modeling choices to benchmark dataset release. First, we present efforts on creating human annotated data, automatically mining data from large unlabeled speech datasets, and adopting pseudo-labeling to produce weakly supervised data. On the modeling, we take advantage of recent advances in applying self-supervised discrete representations as target for prediction in S2ST and show the effectiveness of leveraging additional text supervision from Mandarin, a language similar to Hokkien, in model training. Finally, we release an S2ST benchmark set to facilitate future research in this field. The demo can be found at https://huggingface.co/spaces/facebook/Hokkien_Translation .