Convolutions and Self-Attention: Re-interpreting Relative Positions in Pre-trained Language Models

TL;DR

This paper reveals the equivalence between relative position embeddings in self-attention and dynamic lightweight convolutions, proposing new convolutional methods to enhance Transformer models for natural language processing tasks.

Contribution

It introduces composite attention unifying relative position embeddings with convolutions and demonstrates their effectiveness in improving BERT's performance across tasks.

Findings

Convolutions improve downstream task performance.

Relative position embeddings are equivalent to dynamic lightweight convolutions.

Different convolution types and injection points affect model pre-training.

Abstract

In this paper, we detail the relationship between convolutions and self-attention in natural language tasks. We show that relative position embeddings in self-attention layers are equivalent to recently-proposed dynamic lightweight convolutions, and we consider multiple new ways of integrating convolutions into Transformer self-attention. Specifically, we propose composite attention, which unites previous relative position embedding methods under a convolutional framework. We conduct experiments by training BERT with composite attention, finding that convolutions consistently improve performance on multiple downstream tasks, replacing absolute position embeddings. To inform future work, we present results comparing lightweight convolutions, dynamic convolutions, and depthwise-separable convolutions in language model pre-training, considering multiple injection points for convolutions in self-attention layers.