Why Is Attention Sparse In Particle Transformer?

TL;DR

This paper investigates the origin of sparse attention in Particle Transformer models used for jet tagging, revealing that the sparsity mainly stems from the attention mechanism itself and demonstrating the model's ability to identify key jet features without explicit inputs.

Contribution

The study clarifies that sparse attention in ParT arises primarily from the attention mechanism and shows the model's capacity to detect important jet substructure elements without explicit particle identification.

Findings

Sparse attention mainly originates from the attention mechanism.

ParT can identify key jet features without explicit particle IDs.

Attention sparsity is consistent across multiple datasets.

Abstract

Transformer-based models have achieved state-of-the-art performance in jet tagging at the CERN Large Hadron Collider (LHC), with the Particle Transformer (ParT) representing a leading example of such models. A striking feature of ParT is its sparse, nearly binary, attention structure, raising questions about the origin of this behavior and whether it encodes physically meaningful correlations. In this work, we investigate the source of ParT's sparse attention by comparing models trained on multiple benchmark datasets and examine the relative contributions of the attention term and the physics-inspired interaction matrix before softmax. We find that binary sparsity arises primarily from the attention mechanism itself, with the interaction matrix playing a secondary role. Moreove, we show that ParT is able to identify key jet substructure elements, such as leptons in semileptonic top decays, even without explicit particle identification inputs. These results provide new insight into the interpretability of transformer-based jet taggers and clarify the conditions under which sparse attention patterns emerge in ParT.