It's Not a Walk in the Park! Challenges of Idiom Translation in Speech-to-text Systems

TL;DR

This paper evaluates the challenges of translating idioms in speech-to-text systems, revealing significant performance drops compared to text translation and highlighting the need for specialized strategies.

Contribution

It systematically compares SLT and MT systems on idiom translation across two languages, providing insights into their limitations and areas for improvement.

Findings

SLT systems perform poorly on idioms, often translating literally.

MT systems and LLMs handle idioms better than SLT.

There is a significant gap in idiom translation quality in current SLT architectures.

Abstract

Idioms are defined as a group of words with a figurative meaning not deducible from their individual components. Although modern machine translation systems have made remarkable progress, translating idioms remains a major challenge, especially for speech-to-text systems, where research on this topic is notably sparse. In this paper, we systematically evaluate idiom translation as compared to conventional news translation in both text-to-text machine translation (MT) and speech-to-text translation (SLT) systems across two language pairs (German to English, Russian to English). We compare state-of-the-art end-to-end SLT systems (SeamlessM4T SLT-to-text, Whisper Large v3) with MT systems (SeamlessM4T SLT-to-text, No Language Left Behind), Large Language Models (DeepSeek, LLaMA) and cascaded alternatives. Our results reveal that SLT systems experience a pronounced performance drop on idiomatic data, often reverting to literal translations even in higher layers, whereas MT systems and Large Language Models demonstrate better handling of idioms. These findings underscore the need for idiom-specific strategies and improved internal representations in SLT architectures.