T-Modules: Translation Modules for Zero-Shot Cross-Modal Machine Translation

TL;DR

This paper introduces T-Modules, a novel zero-shot cross-modal translation framework that encodes speech and text into a shared space, enabling translation across languages and modalities without cross-modal labeled data.

Contribution

The paper proposes a new joint representation approach and decoding strategies for zero-shot cross-modal translation, including speech-to-speech and text-to-speech, without requiring cross-modal training data.

Findings

Achieves competitive results on multiple translation tasks.

Significantly improves zero-shot speech translation on Must-C.

First to demonstrate zero-shot speech-to-speech and text-to-speech translation.

Abstract

We present a new approach to perform zero-shot cross-modal transfer between speech and text for translation tasks. Multilingual speech and text are encoded in a joint fixed-size representation space. Then, we compare different approaches to decode these multimodal and multilingual fixed-size representations, enabling zero-shot translation between languages and modalities. All our models are trained without the need of cross-modal labeled translation data. Despite a fixed-size representation, we achieve very competitive results on several text and speech translation tasks. In particular, we significantly improve the state-of-the-art for zero-shot speech translation on Must-C. Incorporating a speech decoder in our framework, we introduce the first results for zero-shot direct speech-to-speech and text-to-speech translation.