CMU's IWSLT 2024 Simultaneous Speech Translation System

TL;DR

This paper presents CMU's end-to-end simultaneous speech translation system for English to German, utilizing WavLM, Llama2, and a two-stage training process, achieving competitive BLEU scores with low latency.

Contribution

The paper introduces a novel end-to-end SST system combining WavLM and Llama2 with a two-stage training approach for streaming speech translation.

Findings

Achieved BLEU score of 31.1 offline

Attained BLEU score of 29.5 at 2 seconds latency

Demonstrated effective adaptation of offline model for SST

Abstract

This paper describes CMU's submission to the IWSLT 2024 Simultaneous Speech Translation (SST) task for translating English speech to German text in a streaming manner. Our end-to-end speech-to-text (ST) system integrates the WavLM speech encoder, a modality adapter, and the Llama2-7B-Base model as the decoder. We employ a two-stage training approach: initially, we align the representations of speech and text, followed by full fine-tuning. Both stages are trained on MuST-c v2 data with cross-entropy loss. We adapt our offline ST model for SST using a simple fixed hold-n policy. Experiments show that our model obtains an offline BLEU score of 31.1 and a BLEU score of 29.5 under 2 seconds latency on the MuST-C-v2 tst-COMMON.