Macro-block dropout for improved regularization in training end-to-end speech recognition models

TL;DR

This paper introduces macro-block dropout, a novel regularization method for end-to-end speech recognition models that applies dropout to macro-blocks of units, leading to significant improvements in Word Error Rates.

Contribution

The paper proposes macro-block dropout, a new regularization technique that applies dropout to large macro-blocks, enhancing model generalization in speech recognition tasks.

Findings

4.30% and 6.13% WER improvements with RNN-T

4.36% and 5.85% WER improvements with AED

Effective regularization for large neural networks

Abstract

This paper proposes a new regularization algorithm referred to as macro-block dropout. The overfitting issue has been a difficult problem in training large neural network models. The dropout technique has proven to be simple yet very effective for regularization by preventing complex co-adaptations during training. In our work, we define a macro-block that contains a large number of units from the input to a Recurrent Neural Network (RNN). Rather than applying dropout to each unit, we apply random dropout to each macro-block. This algorithm has the effect of applying different drop out rates for each layer even if we keep a constant average dropout rate, which has better regularization effects. In our experiments using Recurrent Neural Network-Transducer (RNN-T), this algorithm shows relatively 4.30 % and 6.13 % Word Error Rates (WERs) improvement over the conventional dropout on LibriSpeech test-clean and test-other. With an Attention-based Encoder-Decoder (AED) model, this algorithm shows relatively 4.36 % and 5.85 % WERs improvement over the conventional dropout on the same test sets.