# Analyzing the Structure of Attention in a Transformer Language Model

## TL;DR

This paper investigates the internal attention mechanisms of GPT-2 small, revealing how attention targets different syntactic elements at various layers and captures long-range dependencies, providing insights into model interpretability.

## Contribution

It offers a detailed analysis of attention structures in GPT-2, highlighting how attention aligns with syntax and encodes distant relationships across layers.

## Key findings

- Attention targets different parts of speech at different layers.
- Attention aligns with dependency relations most strongly in middle layers.
- Deep layers capture the most distant relationships.

## Abstract

The Transformer is a fully attention-based alternative to recurrent networks that has achieved state-of-the-art results across a range of NLP tasks. In this paper, we analyze the structure of attention in a Transformer language model, the GPT-2 small pretrained model. We visualize attention for individual instances and analyze the interaction between attention and syntax over a large corpus. We find that attention targets different parts of speech at different layer depths within the model, and that attention aligns with dependency relations most strongly in the middle layers. We also find that the deepest layers of the model capture the most distant relationships. Finally, we extract exemplar sentences that reveal highly specific patterns targeted by particular attention heads.