Isochrony-Controlled Speech-to-Text Translation: A study on translating from Sino-Tibetan to Indo-European Languages

TL;DR

This paper introduces an isochrony-controlled speech-to-text translation model that predicts speech and pause durations to better match source and target speech timing, improving translation length accuracy.

Contribution

It presents a novel duration alignment method that incorporates timing information into the decoder, enhancing length control in speech translation.

Findings

Achieves 0.92 speech overlap on Zh-En test set

Attains 8.9 BLEU score with minimal 1.4 BLEU drop

Improves duration matching compared to previous methods

Abstract

End-to-end speech translation (ST), which translates source language speech directly into target language text, has garnered significant attention in recent years. Many ST applications require strict length control to ensure that the translation duration matches the length of the source audio, including both speech and pause segments. Previous methods often controlled the number of words or characters generated by the Machine Translation model to approximate the source sentence's length without considering the isochrony of pauses and speech segments, as duration can vary between languages. To address this, we present improvements to the duration alignment component of our sequence-to-sequence ST model. Our method controls translation length by predicting the duration of speech and pauses in conjunction with the translation process. This is achieved by providing timing information to the decoder, ensuring it tracks the remaining duration for speech and pauses while generating the translation. The evaluation on the Zh-En test set of CoVoST 2, demonstrates that the proposed Isochrony-Controlled ST achieves 0.92 speech overlap and 8.9 BLEU, which has only a 1.4 BLEU drop compared to the ST baseline.