Speech-to-Speech Translation Pipelines for Conversations in Low-Resource Languages

TL;DR

This paper evaluates speech-to-speech translation pipelines for low-resource languages like Turkish and Pashto, focusing on system performance, component contributions, and the impact of fine-tuning in conversational settings.

Contribution

It introduces a comprehensive evaluation of speech translation pipelines for low-resource languages, including data collection, fine-tuning, and analysis of component independence.

Findings

Best pipeline identified for each language pair and direction.

Component performance ranks are generally independent of other pipeline parts.

Fine-tuning improves translation quality in low-resource settings.

Abstract

The popularity of automatic speech-to-speech translation for human conversations is growing, but the quality varies significantly depending on the language pair. In a context of community interpreting for low-resource languages, namely Turkish and Pashto to/from French, we collected fine-tuning and testing data, and compared systems using several automatic metrics (BLEU, COMET, and BLASER) and human assessments. The pipelines included automatic speech recognition, machine translation, and speech synthesis, with local models and cloud-based commercial ones. Some components have been fine-tuned on our data. We evaluated over 60 pipelines and determined the best one for each direction. We also found that the ranks of components are generally independent of the rest of the pipeline.