Joint Beam Search Integrating CTC, Attention, and Transducer Decoders

TL;DR

This paper introduces a joint 4D modeling approach for end-to-end speech recognition that combines four decoders sharing the same encoder, along with novel joint beam search algorithms, resulting in improved accuracy and robustness.

Contribution

It proposes a unified 4D modeling framework with a two-stage training strategy and three new joint beam search algorithms for enhanced speech recognition performance.

Findings

The 4D model outperforms single-decoder models in accuracy.

Joint beam search algorithms improve decoding performance.

The approach enhances model robustness and efficiency.

Abstract

End-to-end automatic speech recognition (E2E-ASR) can be classified by its decoder architectures, such as connectionist temporal classification (CTC), recurrent neural network transducer (RNN-T), attention-based encoder-decoder, and Mask-CTC models. Each decoder architecture has advantages and disadvantages, leading practitioners to switch between these different models depending on application requirements. Instead of building separate models, we propose a joint modeling scheme where four decoders (CTC, RNN-T, attention, and Mask-CTC) share the same encoder -- we refer to this as 4D modeling. The 4D model is trained jointly, which will bring model regularization and maximize the model robustness thanks to their complementary properties. To efficiently train the 4D model, we introduce a two-stage training strategy that stabilizes the joint training. In addition, we propose three novel joint beam search algorithms by combining three decoders (CTC, RNN-T, and attention) to further improve performance. These three beam search algorithms differ in which decoder is used as the primary decoder. We carefully evaluate the performance and computational tradeoffs associated with each algorithm. Experimental results demonstrate that the jointly trained 4D model outperforms the E2E-ASR models trained with only one individual decoder. Furthermore, we demonstrate that the proposed joint beam search algorithm outperforms the previously proposed CTC/attention decoding.