Direct Speech-to-speech Translation without Textual Annotation using Bottleneck Features

TL;DR

This paper introduces a direct speech-to-speech translation model that eliminates the need for textual annotations by using bottleneck features, achieving comparable performance to traditional cascaded systems.

Contribution

It proposes a novel end-to-end speech translation approach using bottleneck features as intermediate objectives, removing the requirement for textual annotations during training.

Findings

Performance matches cascaded systems in translation quality

Feasibility demonstrated on Mandarin-Cantonese translation

No need for textual annotation or phoneme prediction modules

Abstract

Speech-to-speech translation directly translates a speech utterance to another between different languages, and has great potential in tasks such as simultaneous interpretation. State-of-art models usually contains an auxiliary module for phoneme sequences prediction, and this requires textual annotation of the training dataset. We propose a direct speech-to-speech translation model which can be trained without any textual annotation or content information. Instead of introducing an auxiliary phoneme prediction task in the model, we propose to use bottleneck features as intermediate training objectives for our model to ensure the translation performance of the system. Experiments on Mandarin-Cantonese speech translation demonstrate the feasibility of the proposed approach and the performance can match a cascaded system with respect of translation and synthesis qualities.