CMOT: Cross-modal Mixup via Optimal Transport for Speech Translation

TL;DR

This paper introduces CMOT, a novel method that uses optimal transport to align and mix speech and text sequences at the token level, significantly improving end-to-end speech translation performance.

Contribution

The paper proposes a cross-modal mixup technique using optimal transport to better align speech and text, addressing the modality gap in speech translation.

Findings

Achieves an average BLEU score of 30.0 on MuST-C benchmark

Outperforms previous speech translation methods

Effectively aligns speech and text modalities using optimal transport

Abstract

End-to-end speech translation (ST) is the task of translating speech signals in the source language into text in the target language. As a cross-modal task, end-to-end ST is difficult to train with limited data. Existing methods often try to transfer knowledge from machine translation (MT), but their performances are restricted by the modality gap between speech and text. In this paper, we propose Cross-modal Mixup via Optimal Transport CMOT to overcome the modality gap. We find the alignment between speech and text sequences via optimal transport and then mix up the sequences from different modalities at a token level using the alignment. Experiments on the MuST-C ST benchmark demonstrate that CMOT achieves an average BLEU of 30.0 in 8 translation directions, outperforming previous methods. Further analysis shows CMOT can adaptively find the alignment between modalities, which helps alleviate the modality gap between speech and text. Code is publicly available at https://github.com/ictnlp/CMOT.