Do Attention Heads Compete or Cooperate during Counting?

TL;DR

This paper investigates how attention heads in small transformers collaborate or compete during counting tasks, revealing they act as a pseudo-ensemble with non-uniform aggregation needed for syntactic encoding.

Contribution

It provides the first mechanistic interpretability analysis of attention head interactions during counting in small transformers, highlighting their ensemble-like behavior.

Findings

Attention heads behave as a pseudo-ensemble during counting.

Outputs require non-uniform aggregation for syntactic encoding.

Heads solve subtasks collaboratively rather than competitively.

Abstract

We present an in-depth mechanistic interpretability analysis of training small transformers on an elementary task, counting, which is a crucial deductive step in many algorithms. In particular, we investigate the collaboration/competition among the attention heads: we ask whether the attention heads behave as a pseudo-ensemble, all solving the same subtask, or they perform different subtasks, meaning that they can only solve the original task in conjunction. Our work presents evidence that on the semantics of the counting task, attention heads behave as a pseudo-ensemble, but their outputs need to be aggregated in a non-uniform manner in order to create an encoding that conforms to the syntax. Our source code will be available upon publication.