SimulSeamless: FBK at IWSLT 2024 Simultaneous Speech Translation

TL;DR

This paper introduces SimulSeamless, a novel approach for simultaneous speech translation that combines existing models with a new policy, enabling effective translation across many languages without retraining.

Contribution

The paper presents SimulSeamless, a new method integrating AlignAtt with SeamlessM4T for multilingual simultaneous speech translation without retraining.

Findings

Achieved comparable or better results than previous submissions.

Supported over 143 source and 200 target languages.

Released the model for public use.

Abstract

This paper describes the FBK's participation in the Simultaneous Translation Evaluation Campaign at IWSLT 2024. For this year's submission in the speech-to-text translation (ST) sub-track, we propose SimulSeamless, which is realized by combining AlignAtt and SeamlessM4T in its medium configuration. The SeamlessM4T model is used "off-the-shelf" and its simultaneous inference is enabled through the adoption of AlignAtt, a SimulST policy based on cross-attention that can be applied without any retraining or adaptation of the underlying model for the simultaneous task. We participated in all the Shared Task languages (English->{German, Japanese, Chinese}, and Czech->English), achieving acceptable or even better results compared to last year's submissions. SimulSeamless, covering more than 143 source languages and 200 target languages, is released at: https://github.com/hlt-mt/FBK-fairseq/.