Enhancing End-to-End Conversational Speech Translation Through Target Language Context Utilization

TL;DR

This paper introduces a novel approach to end-to-end conversational speech translation by incorporating target language context, speaker info, and context dropout, significantly improving translation coherence and robustness in extended audio segments.

Contribution

It presents the first integration of target language context in E2E speech translation, along with context dropout and speaker info, to enhance translation quality and robustness.

Findings

Contextual E2E-ST outperforms isolated utterance-based methods.

Context primarily helps with style, anaphora, and named entity resolution.

Adding speaker info further improves translation coherence.

Abstract

Incorporating longer context has been shown to benefit machine translation, but the inclusion of context in end-to-end speech translation (E2E-ST) remains under-studied. To bridge this gap, we introduce target language context in E2E-ST, enhancing coherence and overcoming memory constraints of extended audio segments. Additionally, we propose context dropout to ensure robustness to the absence of context, and further improve performance by adding speaker information. Our proposed contextual E2E-ST outperforms the isolated utterance-based E2E-ST approach. Lastly, we demonstrate that in conversational speech, contextual information primarily contributes to capturing context style, as well as resolving anaphora and named entities.