Fast-MD: Fast Multi-Decoder End-to-End Speech Translation with Non-Autoregressive Hidden Intermediates

TL;DR

Fast-MD introduces a non-autoregressive decoding approach for multi-decoder speech translation, significantly improving inference speed while maintaining translation quality, making it more suitable for real-world applications.

Contribution

It proposes a novel non-autoregressive hidden intermediate generation method for multi-decoder speech translation, reducing decoding time without sacrificing accuracy.

Findings

Achieved 2x faster decoding on GPU and 4x on CPU compared to naive MD.

Maintained comparable translation quality with faster inference.

Enhanced model performance with Conformer encoder and intermediate CTC loss.

Abstract

The multi-decoder (MD) end-to-end speech translation model has demonstrated high translation quality by searching for better intermediate automatic speech recognition (ASR) decoder states as hidden intermediates (HI). It is a two-pass decoding model decomposing the overall task into ASR and machine translation sub-tasks. However, the decoding speed is not fast enough for real-world applications because it conducts beam search for both sub-tasks during inference. We propose Fast-MD, a fast MD model that generates HI by non-autoregressive (NAR) decoding based on connectionist temporal classification (CTC) outputs followed by an ASR decoder. We investigated two types of NAR HI: (1) parallel HI by using an autoregressive Transformer ASR decoder and (2) masked HI by using Mask-CTC, which combines CTC and the conditional masked language model. To reduce a mismatch in the ASR decoder between teacher-forcing during training and conditioning on CTC outputs during testing, we also propose sampling CTC outputs during training. Experimental evaluations on three corpora show that Fast-MD achieved about 2x and 4x faster decoding speed than that of the na\"ive MD model on GPU and CPU with comparable translation quality. Adopting the Conformer encoder and intermediate CTC loss further boosts its quality without sacrificing decoding speed.