Optimizing Rare Word Accuracy in Direct Speech Translation with a Retrieval-and-Demonstration Approach

TL;DR

This paper introduces a retrieval-and-demonstration method to improve rare word translation accuracy in direct speech translation models by leveraging retrieved examples, significantly enhancing performance especially with gold examples.

Contribution

It proposes a novel retrieval-and-demonstration approach for direct speech translation that effectively incorporates retrieved examples to improve rare word translation accuracy.

Findings

Improved rare word translation accuracy by 17.6% with gold examples

Retrieval-based approach outperforms other modalities in robustness

Effective adaptation of standard models to leverage examples for rare words

Abstract

Direct speech translation (ST) models often struggle with rare words. Incorrect translation of these words can have severe consequences, impacting translation quality and user trust. While rare word translation is inherently challenging for neural models due to sparse learning signals, real-world scenarios often allow access to translations of past recordings on similar topics. To leverage these valuable resources, we propose a retrieval-and-demonstration approach to enhance rare word translation accuracy in direct ST models. First, we adapt existing ST models to incorporate retrieved examples for rare word translation, which allows the model to benefit from prepended examples, similar to in-context learning. We then develop a cross-modal (speech-to-speech, speech-to-text, text-to-text) retriever to locate suitable examples. We demonstrate that standard ST models can be effectively adapted to leverage examples for rare word translation, improving rare word translation accuracy over the baseline by 17.6% with gold examples and 8.5% with retrieved examples. Moreover, our speech-to-speech retrieval approach outperforms other modalities and exhibits higher robustness to unseen speakers. Our code is publicly available (https://github.com/SiqiLii/Retrieve-and-Demonstration-ST).