The Conformer Encoder May Reverse the Time Dimension

TL;DR

This paper investigates how Conformer encoders can reverse the sequence in time during training, analyzes the underlying mechanisms, and proposes methods to prevent this reversal while also deriving label-frame alignments from gradients.

Contribution

It reveals the sequence reversal phenomenon in Conformer encoders, analyzes its causes, and introduces techniques to avoid it and extract alignments using gradient information.

Findings

Conformer encoders can reverse sequence order during training.

Self-attention dominance leads to reversed information flow.

Gradient-based methods can obtain label-frame alignments.

Abstract

We sometimes observe monotonically decreasing cross-attention weights in our Conformer-based global attention-based encoder-decoder (AED) models, Further investigation shows that the Conformer encoder reverses the sequence in the time dimension. We analyze the initial behavior of the decoder cross-attention mechanism and find that it encourages the Conformer encoder self-attention to build a connection between the initial frames and all other informative frames. Furthermore, we show that, at some point in training, the self-attention module of the Conformer starts dominating the output over the preceding feed-forward module, which then only allows the reversed information to pass through. We propose methods and ideas of how this flipping can be avoided and investigate a novel method to obtain label-frame-position alignments by using the gradients of the label log probabilities w.r.t. the encoder input frames.