Improving Direct Persian-English Speech-to-Speech Translation with Discrete Units and Synthetic Parallel Data

TL;DR

This paper introduces a direct Persian-English speech-to-speech translation system that leverages synthetic parallel data, discrete speech units, and self-supervised pre-training to improve translation quality in low-resource scenarios.

Contribution

It presents a novel pipeline combining discrete units and synthetic data generation to enhance direct S2ST for low-resource languages like Persian-English.

Findings

Achieved 4.6 BLEU improvement over baselines.

Constructed a sixfold larger parallel speech corpus.

Demonstrated effectiveness of synthetic data in low-resource S2ST.

Abstract

Direct speech-to-speech translation (S2ST), in which all components are trained jointly, is an attractive alternative to cascaded systems because it offers a simpler pipeline and lower inference latency. However, direct S2ST models require large amounts of parallel speech data in the source and target languages, which are rarely available for low-resource languages such as Persian. This paper presents a direct S2ST system for translating Persian speech into English speech, as well as a pipeline for synthetic parallel Persian-English speech generation. The model comprises three components: (1) a conformer-based encoder, initialized from self-supervised pre-training, maps source speech to high-level acoustic representations; (2) a causal transformer decoder with relative position multi-head attention translates these representations into discrete target speech units; (3) a unit-based neural vocoder generates waveforms from the predicted discrete units. To mitigate the data scarcity problem, we construct a new Persian-English parallel speech corpus by translating Persian speech transcriptions into English using a large language model and then synthesizing the corresponding English speech with a state-of-the-art zero-shot text-to-speech system. The resulting corpus increases the amount of available parallel speech by roughly a factor of six. On the Persian-English portion of the CVSS corpus, the proposed model achieves improvement of 4.6 ASR BLEU with the synthetic data over direct baselines. These results indicate that combining self-supervised pre-training, discrete speech units, and synthetic parallel data is effective for improving direct S2ST in low-resource language pairs such as Persian-English