Timbre-Aware LLM-based Direct Speech-to-Speech Translation Extendable to Multiple Language Pairs

TL;DR

This paper introduces DS2ST-LM, a scalable, multilingual direct speech-to-speech translation framework that leverages a large language model and timbre control to improve translation quality, stability, and speaker identity preservation across multiple languages.

Contribution

The work presents a novel, scalable direct S2ST system integrating a multilingual LLM, synthetic data augmentation, and timbre-aware synthesis, advancing multilingual capabilities and speaker preservation.

Findings

Outperforms traditional cascaded and baseline systems in BLEU, METEOR, BLEURT, and COMET metrics.

Effectively extends to multiple languages including French, Spanish, German, Hindi, Bengali, and Urdu.

Timbre-aware synthesis improves speaker similarity and naturalness.

Abstract

Direct Speech-to-Speech Translation (S2ST) has gained increasing attention for its ability to translate speech from one language to another, while reducing error propagation and latency inherent in traditional cascaded pipelines. However, existing direct S2ST systems continue to face notable challenges, including instability in semantic-acoustic alignment when parallel speech data is scarce, difficulty in preserving speaker identity, and limited multilingual scalability. In this work, we introduce DS2ST-LM, a scalable, single-stage direct S2ST framework leveraging a multilingual Large Language Model (LLM). The architecture integrates a Whisper speech encoder, a learnable projection module, a Qwen2-0.5B LLM, and a timbre-controlled vocoder. We construct GigaS2S-1000, a 1000-hour bilingual corpus by extending the GigaST dataset with high-fidelity synthetic target speech, and show that this synthetic data alleviates data scarcity to some extent. We investigate two semantic token generation strategies: speech-derived S3 tokens and text-derived tokens generated by a pre-trained LLM, and analyze their impact on training stability and semantic consistency. We further evaluate three projection architectures (Linear, Conv1D-Linear, and Q-Former) and observe that while higher-capacity projectors converge faster, the simple Linear projector achieves higher performance. Extensive experiments demonstrate that DS2ST-LM outperforms traditional cascaded and ST (Qwen-Audio) + TTS baselines across both lexical (BLEU, METEOR) and semantic (BLEURT, COMET) metrics, while extending to multiple language pairs, including French, Spanish, German, Hindi, Bengali, and Urdu. Furthermore, we incorporate timbre-aware speech synthesis to preserve speaker information, enabling DS2ST-LM to surpass prior direct S2ST systems in both speaker similarity and perceptual naturalness.