RASST: Fast Cross-modal Retrieval-Augmented Simultaneous Speech Translation

TL;DR

RASST introduces a fast, integrated cross-modal retrieval method to enhance terminology accuracy and overall quality in simultaneous speech translation, addressing challenges with domain-specific terms.

Contribution

It presents a novel retrieval-augmented pipeline for SST that trains a lightweight retriever and synthesizes training data to improve terminology translation.

Findings

Terminology translation accuracy improved by up to 16%.

Overall translation quality increased by up to 3 BLEU points.

Effective integration of retrieval enhances SST performance.

Abstract

Simultaneous speech translation (SST) produces target text incrementally from partial speech input. Recent speech large language models (Speech LLMs) have substantially improved SST quality, yet they still struggle to correctly translate rare and domain-specific terminology. While retrieval augmentation has been effective for terminology translation in machine translation, bringing retrieval to SST is non-trivial: it requires fast and accurate cross-modal (speech-to-text) retrieval under partial, continually arriving input, and the model must decide whether and when to apply retrieved terms during incremental generation. We propose Retrieval-Augmented Simultaneous Speech Translation (RASST), which tightly integrates cross-modal retrieval into the SST pipeline. RASST trains a lightweight speech-text retriever and performs efficient sliding-window retrieval, providing chunkwise terminology hints to the Speech LLM. We further synthesize training data that teaches the Speech LLM to leverage retrieved terms precisely. Experiments on three language directions of the ACL 60/60 dev set show that RASST improves terminology translation accuracy by up to 16% and increases overall translation quality by up to 3 BLEU points, with ablations confirming the contribution of each component.