Analysis of Self-Attention Head Diversity for Conformer-based Automatic Speech Recognition

TL;DR

This paper analyzes the diversity of attention heads in Conformer-based speech recognition, finding that promoting diversity improves accuracy and correlates with gradient similarity, with auxiliary loss functions being most effective.

Contribution

It introduces methods to increase attention head diversity in Conformer models and demonstrates that diversity-promoting auxiliary losses lead to significant WER improvements.

Findings

Attention heads become highly correlated during training

Diversity-promoting auxiliary losses improve WER by up to 6%

Head diversity correlates with gradient similarity

Abstract

Attention layers are an integral part of modern end-to-end automatic speech recognition systems, for instance as part of the Transformer or Conformer architecture. Attention is typically multi-headed, where each head has an independent set of learned parameters and operates on the same input feature sequence. The output of multi-headed attention is a fusion of the outputs from the individual heads. We empirically analyze the diversity between representations produced by the different attention heads and demonstrate that the heads become highly correlated during the course of training. We investigate a few approaches to increasing attention head diversity, including using different attention mechanisms for each head and auxiliary training loss functions to promote head diversity. We show that introducing diversity-promoting auxiliary loss functions during training is a more effective approach, and obtain WER improvements of up to 6% relative on the Librispeech corpus. Finally, we draw a connection between the diversity of attention heads and the similarity of the gradients of head parameters.