Guiding Attention for Self-Supervised Learning with Transformers

TL;DR

This paper introduces a simple auxiliary loss to guide attention in Transformers, improving training efficiency and downstream task performance, especially in low-resource scenarios, by leveraging non-linguistic attention patterns.

Contribution

It presents a novel, computationally efficient method to guide attention in self-supervised Transformer training, enhancing convergence and performance.

Findings

Faster convergence of models with the proposed method

Improved performance on downstream tasks

State-of-the-art results in low-resource settings

Abstract

In this paper, we propose a simple and effective technique to allow for efficient self-supervised learning with bi-directional Transformers. Our approach is motivated by recent studies demonstrating that self-attention patterns in trained models contain a majority of non-linguistic regularities. We propose a computationally efficient auxiliary loss function to guide attention heads to conform to such patterns. Our method is agnostic to the actual pre-training objective and results in faster convergence of models as well as better performance on downstream tasks compared to the baselines, achieving state of the art results in low-resource settings. Surprisingly, we also find that linguistic properties of attention heads are not necessarily correlated with language modeling performance.