Cross-Modal Robustness Transfer (CMRT): Training Robust Speech Translation Models Using Adversarial Text

TL;DR

This paper introduces CMRT, a novel framework that enhances the robustness of speech translation models against morphological adversarial attacks by transferring text-based adversarial robustness to speech, without needing adversarial speech data.

Contribution

The paper proposes Cross-Modal Robustness Transfer (CMRT), a new method to improve speech translation model robustness by leveraging text adversarial training, avoiding costly adversarial speech data generation.

Findings

CMRT improves adversarial robustness by over 3 BLEU points on average.

The method establishes a new baseline for robust speech translation.

It effectively transfers robustness from text to speech modality.

Abstract

End-to-End Speech Translation (E2E-ST) has seen significant advancements, yet current models are primarily benchmarked on curated, "clean" datasets. This overlooks critical real-world challenges, such as morphological robustness to inflectional variations common in non-native or dialectal speech. In this work, we adapt a text-based adversarial attack targeting inflectional morphology to the speech domain and demonstrate that state-of-the-art E2E-ST models are highly vulnerable it. While adversarial training effectively mitigates such risks in text-based tasks, generating high-quality adversarial speech data remains computationally expensive and technically challenging. To address this, we propose Cross-Modal Robustness Transfer (CMRT), a framework that transfers adversarial robustness from the text modality to the speech modality. Our method eliminates the requirement for adversarial speech data during training. Extensive experiments across four language pairs demonstrate that CMRT improves adversarial robustness by an average of more than 3 BLEU points, establishing a new baseline for robust E2E-ST without the overhead of generating adversarial speech.