CrossVoice: Crosslingual Prosody Preserving Cascade-S2ST using Transfer Learning

TL;DR

CrossVoice introduces a cascade-based multilingual S2ST system that leverages transfer learning to preserve prosody, achieving high translation quality and naturalness comparable to human speech.

Contribution

It presents a novel cascade S2ST system with cross-lingual prosody transfer using transfer learning, outperforming direct systems in quality and prosody preservation.

Findings

Improved BLEU scores on Fisher Es-En and VoxPopuli Fr-En datasets.

High mean opinion score of 3.75 out of 4 for synthesized speech.

Effective cross-lingual prosody transfer demonstrated on benchmark datasets.

Abstract

This paper presents CrossVoice, a novel cascade-based Speech-to-Speech Translation (S2ST) system employing advanced ASR, MT, and TTS technologies with cross-lingual prosody preservation through transfer learning. We conducted comprehensive experiments comparing CrossVoice with direct-S2ST systems, showing improved BLEU scores on tasks such as Fisher Es-En, VoxPopuli Fr-En and prosody preservation on benchmark datasets CVSS-T and IndicTTS. With an average mean opinion score of 3.75 out of 4, speech synthesized by CrossVoice closely rivals human speech on the benchmark, highlighting the efficacy of cascade-based systems and transfer learning in multilingual S2ST with prosody transfer.