Talking-Heads Attention

TL;DR

Talking-heads attention is a novel variation of multi-head attention that incorporates linear projections across attention heads, improving language modeling and transfer learning performance with minimal additional complexity.

Contribution

It introduces talking-heads attention, a new multi-head attention mechanism with linear projections across heads, enhancing model performance with few extra parameters.

Findings

Improves perplexities on masked language modeling tasks

Enhances transfer-learning quality for language comprehension

Adds minimal parameters and computation

Abstract

We introduce "talking-heads attention" - a variation on multi-head attention which includes linearprojections across the attention-heads dimension, immediately before and after the softmax operation.While inserting only a small number of additional parameters and a moderate amount of additionalcomputation, talking-heads attention leads to better perplexities on masked language modeling tasks, aswell as better quality when transfer-learning to language comprehension and question answering tasks.