Improving Generalization of Transformer for Speech Recognition with Parallel Schedule Sampling and Relative Positional Embedding

TL;DR

This paper introduces parallel scheduled sampling and relative positional embedding to improve the generalization of Transformer models in speech recognition, especially for longer sequences, resulting in significant performance gains.

Contribution

It proposes novel methods—parallel scheduled sampling and relative positional embedding—to enhance Transformer generalization in speech recognition tasks.

Findings

7% relative improvement on short utterances

70% relative gain on long utterances

Effective for a 10,000-hour Mandarin ASR task

Abstract

Transformer has shown promising results in many sequence to sequence transformation tasks recently. It utilizes a number of feed-forward self-attention layers to replace the recurrent neural networks (RNN) in attention-based encoder decoder (AED) architecture. Self-attention layer learns temporal dependence by incorporating sinusoidal positional embedding of tokens in a sequence for parallel computing. Quicker iteration speed in training than sequential operation of RNN can be obtained. Deeper layers of the transformer also make it perform better than RNN-based AED. However, this parallelization ability is lost when applying scheduled sampling training. Self-attention with sinusoidal positional embedding may cause performance degradations for longer sequences that have similar acoustic or semantic information at different positions as well. To address these problems, we propose to use parallel scheduled sampling (PSS) and relative positional embedding (RPE) to help the transformer generalize to unseen data. Our proposed methods achieve a 7% relative improvement for short utterances and a 70% relative gain for long utterances on a 10,000-hour Mandarin ASR task.