The NPU-MSXF Speech-to-Speech Translation System for IWSLT 2023 Speech-to-Speech Translation Task

TL;DR

This paper presents a cascaded speech-to-speech translation system for IWSLT 2023 that effectively handles multi-source input, noisy transcripts, and produces natural, speaker-consistent Chinese speech from English input.

Contribution

The system introduces robust multi-source handling, a three-stage fine-tuning strategy, and a two-stage TTS framework with speaker transfer, advancing speech translation quality and robustness.

Findings

High translation accuracy and speech naturalness achieved.

Demonstrates robustness to multi-source and noisy input.

Effective speaker timbre transfer in translated speech.

Abstract

This paper describes the NPU-MSXF system for the IWSLT 2023 speech-to-speech translation (S2ST) task which aims to translate from English speech of multi-source to Chinese speech. The system is built in a cascaded manner consisting of automatic speech recognition (ASR), machine translation (MT), and text-to-speech (TTS). We make tremendous efforts to handle the challenging multi-source input. Specifically, to improve the robustness to multi-source speech input, we adopt various data augmentation strategies and a ROVER-based score fusion on multiple ASR model outputs. To better handle the noisy ASR transcripts, we introduce a three-stage fine-tuning strategy to improve translation accuracy. Finally, we build a TTS model with high naturalness and sound quality, which leverages a two-stage framework, using network bottleneck features as a robust intermediate representation for speaker timbre and linguistic content disentanglement. Based on the two-stage framework, pre-trained speaker embedding is leveraged as a condition to transfer the speaker timbre in the source English speech to the translated Chinese speech. Experimental results show that our system has high translation accuracy, speech naturalness, sound quality, and speaker similarity. Moreover, it shows good robustness to multi-source data.