Cascade versus Direct Speech Translation: Do the Differences Still Make a Difference?

TL;DR

This paper systematically compares cascade and direct speech translation systems across multiple languages, concluding that their performance gap is now closed and differences are indistinguishable to humans.

Contribution

It provides a comprehensive, multi-faceted evaluation demonstrating that the performance gap between cascade and direct speech translation approaches has been effectively bridged.

Findings

Performance gap between paradigms is closed

Humans cannot distinguish or prefer one approach over the other

Systematic comparison on a public benchmark with professional annotations

Abstract

Five years after the first published proofs of concept, direct approaches to speech translation (ST) are now competing with traditional cascade solutions. In light of this steady progress, can we claim that the performance gap between the two is closed? Starting from this question, we present a systematic comparison between state-of-the-art systems representative of the two paradigms. Focusing on three language directions (English-German/Italian/Spanish), we conduct automatic and manual evaluations, exploiting high-quality professional post-edits and annotations. Our multi-faceted analysis on one of the few publicly available ST benchmarks attests for the first time that: i) the gap between the two paradigms is now closed, and ii) the subtle differences observed in their behavior are not sufficient for humans neither to distinguish them nor to prefer one over the other.