SLM-S2ST: A multimodal language model for direct speech-to-speech translation

TL;DR

This paper introduces SLM-S2ST, a multimodal language model capable of direct speech-to-speech translation, leveraging an audio transformer and vocoder to produce high-quality translated speech, outperforming existing models on benchmark datasets.

Contribution

The paper presents SLM-S2ST, a novel multimodal model that extends previous speech-aware language models to directly generate translated speech using an audio transformer and vocoder.

Findings

SLM-S2ST outperforms baseline models on CVSS-C dataset.

Scaling data and model size achieves SOTA performance.

Efficient speech-to-speech translation with high quality.

Abstract

Speech-aware language models (LMs) have demonstrated capabilities in understanding spoken language while generating text-based responses. However, enabling them to produce speech output efficiently and effectively remains a challenge. In this paper, we present SLM-S2ST, a multimodal LM for direct speech-to-speech translation (S2ST), built on the open-source Phi4-MM model. SLM-S2ST extends its predecessor by generating translated speech using an audio transformer head that predicts audio tokens with a delay relative to text tokens, followed by a streaming vocoder for waveform synthesis. Our experimental results on the CVSS-C dataset demonstrate SLM-S2ST's superior performance, significantly surpassing existing baseline models trained on the same dataset. Furthermore, when we scale up the training data and the model size, SLM-S2ST reaches on-par performance with the current SOTA model.