Consistent Transcription and Translation of Speech

TL;DR

This paper investigates methods for jointly transcribing and translating speech, emphasizing the importance of output consistency and comparing traditional and end-to-end models to improve user experience.

Contribution

It introduces a methodology to evaluate and enhance the consistency between transcription and translation outputs in speech processing models.

Findings

End-to-end models with coupled inference achieve better consistency.

Direct models are less suitable for joint transcription and translation.

Optimizing for consistency involves trade-offs with accuracy.

Abstract

The conventional paradigm in speech translation starts with a speech recognition step to generate transcripts, followed by a translation step with the automatic transcripts as input. To address various shortcomings of this paradigm, recent work explores end-to-end trainable direct models that translate without transcribing. However, transcripts can be an indispensable output in practical applications, which often display transcripts alongside the translations to users. We make this common requirement explicit and explore the task of jointly transcribing and translating speech. While high accuracy of transcript and translation are crucial, even highly accurate systems can suffer from inconsistencies between both outputs that degrade the user experience. We introduce a methodology to evaluate consistency and compare several modeling approaches, including the traditional cascaded approach and end-to-end models. We find that direct models are poorly suited to the joint transcription/translation task, but that end-to-end models that feature a coupled inference procedure are able to achieve strong consistency. We further introduce simple techniques for directly optimizing for consistency, and analyze the resulting trade-offs between consistency, transcription accuracy, and translation accuracy.