Mask CTC: Non-Autoregressive End-to-End ASR with CTC and Mask Predict

TL;DR

Mask CTC introduces a non-autoregressive speech recognition model that refines CTC outputs through mask prediction, significantly reducing inference time while maintaining high accuracy.

Contribution

The paper proposes Mask CTC, a novel non-autoregressive end-to-end ASR framework that combines CTC with mask prediction for faster inference and competitive accuracy.

Findings

Outperforms standard CTC with lower WER (e.g., 17.9% to 12.1%)

Achieves near-autoregressive accuracy with much faster inference (0.07 RTF)

Effective on multiple speech recognition tasks.

Abstract

We present Mask CTC, a novel non-autoregressive end-to-end automatic speech recognition (ASR) framework, which generates a sequence by refining outputs of the connectionist temporal classification (CTC). Neural sequence-to-sequence models are usually \textit{autoregressive}: each output token is generated by conditioning on previously generated tokens, at the cost of requiring as many iterations as the output length. On the other hand, non-autoregressive models can simultaneously generate tokens within a constant number of iterations, which results in significant inference time reduction and better suits end-to-end ASR model for real-world scenarios. In this work, Mask CTC model is trained using a Transformer encoder-decoder with joint training of mask prediction and CTC. During inference, the target sequence is initialized with the greedy CTC outputs and low-confidence tokens are masked based on the CTC probabilities. Based on the conditional dependence between output tokens, these masked low-confidence tokens are then predicted conditioning on the high-confidence tokens. Experimental results on different speech recognition tasks show that Mask CTC outperforms the standard CTC model (e.g., 17.9% -> 12.1% WER on WSJ) and approaches the autoregressive model, requiring much less inference time using CPUs (0.07 RTF in Python implementation). All of our codes will be publicly available.