Sequence-to-Sequence Learning via Attention Transfer for Incremental Speech Recognition

TL;DR

This paper proposes a novel incremental speech recognition method using attention transfer, enabling real-time transcription by mimicking full-utterance models with shorter input sequences, achieving low delay with comparable accuracy.

Contribution

It introduces a way to adapt attention-based sequence-to-sequence ASR for incremental recognition by using a teacher-student framework with attention transfer, maintaining original architecture.

Findings

Achieves low delay of about 1.7 seconds in recognition start.

Maintains comparable accuracy to full-utterance models.

Simplifies incremental speech recognition without complex new frameworks.

Abstract

Attention-based sequence-to-sequence automatic speech recognition (ASR) requires a significant delay to recognize long utterances because the output is generated after receiving entire input sequences. Although several studies recently proposed sequence mechanisms for incremental speech recognition (ISR), using different frameworks and learning algorithms is more complicated than the standard ASR model. One main reason is because the model needs to decide the incremental steps and learn the transcription that aligns with the current short speech segment. In this work, we investigate whether it is possible to employ the original architecture of attention-based ASR for ISR tasks by treating a full-utterance ASR as the teacher model and the ISR as the student model. We design an alternative student network that, instead of using a thinner or a shallower model, keeps the original architecture of the teacher model but with shorter sequences (few encoder and decoder states). Using attention transfer, the student network learns to mimic the same alignment between the current input short speech segments and the transcription. Our experiments show that by delaying the starting time of recognition process with about 1.7 sec, we can achieve comparable performance to one that needs to wait until the end.