Hearing to Translate: The Effectiveness of Speech Modality Integration into LLMs

TL;DR

This paper benchmarks SpeechLLMs against traditional systems for speech translation, finding that while cascades are generally reliable, recent SpeechLLMs can match or outperform them in certain conditions, emphasizing the importance of LLM integration.

Contribution

It introduces Hearing to Translate, the first comprehensive benchmark suite evaluating SpeechLLMs against strong baselines across diverse conditions and languages.

Findings

Cascaded systems are generally more reliable overall.

Recent SpeechLLMs can match or outperform cascades in various settings.

Integrating an LLM is crucial for high-quality speech translation.

Abstract

As Large Language Models (LLMs) expand beyond text, integrating speech as a native modality has given rise to SpeechLLMs, which directly process spoken language and enable speech-to-text translation (ST) and other downstream tasks, bypassing traditional transcription-based pipelines. Whether this integration improves ST quality over established cascaded architectures, however, remains an open question. We present Hearing to Translate, the first comprehensive test suite rigorously benchmarking 6 state-of-the-art SpeechLLMs against 16 strong direct and cascade systems that couple leading speech foundation models (SFM), with multilingual LLMs. Our analysis spans 16 benchmarks, 13 language pairs, and 9 challenging conditions, including disfluent, noisy, and long-form speech. Across this extensive evaluation, we find that cascaded systems remain the most reliable solution overall, but most recent SpeechLLMs can match or even outperform cascades in various settings while SFMs lag behind both, highlighting that integrating an LLM, either within the model or in a pipeline, is essential for high-quality speech translation.