CASS-NAT: CTC Alignment-based Single Step Non-autoregressive Transformer for Speech Recognition

TL;DR

This paper introduces CASS-NAT, a non-autoregressive transformer for speech recognition that uses CTC alignment to enable parallel decoding, achieving competitive accuracy with significantly faster inference.

Contribution

The paper presents a novel CTC alignment-based approach for single-step non-autoregressive speech recognition, replacing word embeddings with token-level acoustic embeddings for parallel decoding.

Findings

Achieves 3.8% WER on Librispeech test clean

Runs 51.2x faster than autoregressive baselines

Potential WER of 2.3% with oracle alignment

Abstract

We propose a CTC alignment-based single step non-autoregressive transformer (CASS-NAT) for speech recognition. Specifically, the CTC alignment contains the information of (a) the number of tokens for decoder input, and (b) the time span of acoustics for each token. The information are used to extract acoustic representation for each token in parallel, referred to as token-level acoustic embedding which substitutes the word embedding in autoregressive transformer (AT) to achieve parallel generation in decoder. During inference, an error-based alignment sampling method is proposed to be applied to the CTC output space, reducing the WER and retaining the parallelism as well. Experimental results show that the proposed method achieves WERs of 3.8%/9.1% on Librispeech test clean/other dataset without an external LM, and a CER of 5.8% on Aishell1 Mandarin corpus, respectively1. Compared to the AT baseline, the CASS-NAT has a performance reduction on WER, but is 51.2x faster in terms of RTF. When decoding with an oracle CTC alignment, the lower bound of WER without LM reaches 2.3% on the test-clean set, indicating the potential of the proposed method.