Translatotron 2: High-quality direct speech-to-speech translation with voice preservation

TL;DR

Translatotron 2 is an end-to-end neural speech-to-speech translation model that significantly improves translation and speech quality while effectively preserving speaker voices without segmentation, enhancing privacy and reducing misuse.

Contribution

It introduces a novel end-to-end speech translation model with a simple voice preservation method that does not require speaker segmentation, outperforming previous models.

Findings

Outperforms Translatotron with up to +15.5 BLEU score improvement.

Achieves translation quality comparable to cascade systems.

Effectively preserves speaker voices and privacy without segmentation.

Abstract

We present Translatotron 2, a neural direct speech-to-speech translation model that can be trained end-to-end. Translatotron 2 consists of a speech encoder, a linguistic decoder, an acoustic synthesizer, and a single attention module that connects them together. Experimental results on three datasets consistently show that Translatotron 2 outperforms the original Translatotron by a large margin on both translation quality (up to +15.5 BLEU) and speech generation quality, and approaches the same of cascade systems. In addition, we propose a simple method for preserving speakers' voices from the source speech to the translation speech in a different language. Unlike existing approaches, the proposed method is able to preserve each speaker's voice on speaker turns without requiring for speaker segmentation. Furthermore, compared to existing approaches, it better preserves speaker's privacy and mitigates potential misuse of voice cloning for creating spoofing audio artifacts.