Minimum Latency Training of Sequence Transducers for Streaming End-to-End Speech Recognition

TL;DR

This paper introduces a novel training method for sequence transducers in streaming speech recognition that explicitly models and minimizes latency, achieving significant latency reduction with minimal accuracy loss.

Contribution

The paper proposes a new training approach that explicitly incorporates latency modeling into sequence transducer training, improving latency-accuracy trade-offs.

Findings

Reduces latency from 220 ms to 27 ms with only 0.7% WER increase

Outperforms FastEmit and alignment-restricted training in latency reduction

Effective latency reduction demonstrated on WSJ dataset

Abstract

Sequence transducers, such as the RNN-T and the Conformer-T, are one of the most promising models of end-to-end speech recognition, especially in streaming scenarios where both latency and accuracy are important. Although various methods, such as alignment-restricted training and FastEmit, have been studied to reduce the latency, latency reduction is often accompanied with a significant degradation in accuracy. We argue that this suboptimal performance might be caused because none of the prior methods explicitly model and reduce the latency. In this paper, we propose a new training method to explicitly model and reduce the latency of sequence transducer models. First, we define the expected latency at each diagonal line on the lattice, and show that its gradient can be computed efficiently within the forward-backward algorithm. Then we augment the transducer loss with this expected latency, so that an optimal trade-off between latency and accuracy is achieved. Experimental results on the WSJ dataset show that the proposed minimum latency training reduces the latency of causal Conformer-T from 220 ms to 27 ms within a WER degradation of 0.7%, and outperforms conventional alignment-restricted training (110 ms) and FastEmit (67 ms) methods.