Non-autoregressive Transformer with Unified Bidirectional Decoder for Automatic Speech Recognition

TL;DR

This paper introduces NAT-UBD, a non-autoregressive transformer with a unified bidirectional decoder for speech recognition, effectively utilizing both left-to-right and right-to-left contexts without increasing model complexity, and achieving state-of-the-art results.

Contribution

The paper proposes a novel NAT-UBD model with a unified bidirectional decoder and a special attention mask to prevent information leakage, enhancing speech recognition accuracy and speed.

Findings

Achieves 5.0%/5.5% CER on Aishell1 dev/test sets.

Runs 49.8x faster than autoregressive transformer.

Outperforms previous NAR transformer models.

Abstract

Non-autoregressive (NAR) transformer models have been studied intensively in automatic speech recognition (ASR), and a substantial part of NAR transformer models is to use the casual mask to limit token dependencies. However, the casual mask is designed for the left-to-right decoding process of the non-parallel autoregressive (AR) transformer, which is inappropriate for the parallel NAR transformer since it ignores the right-to-left contexts. Some models are proposed to utilize right-to-left contexts with an extra decoder, but these methods increase the model complexity. To tackle the above problems, we propose a new non-autoregressive transformer with a unified bidirectional decoder (NAT-UBD), which can simultaneously utilize left-to-right and right-to-left contexts. However, direct use of bidirectional contexts will cause information leakage, which means the decoder output can be affected by the character information from the input of the same position. To avoid information leakage, we propose a novel attention mask and modify vanilla queries, keys, and values matrices for NAT-UBD. Experimental results verify that NAT-UBD can achieve character error rates (CERs) of 5.0%/5.5% on the Aishell1 dev/test sets, outperforming all previous NAR transformer models. Moreover, NAT-UBD can run 49.8x faster than the AR transformer baseline when decoding in a single step.