Exclusive Self Attention

TL;DR

Exclusive Self Attention (XSA) modifies standard self attention by excluding a token's own information, leading to improved language modeling performance, especially on longer sequences, across various model sizes.

Contribution

This paper introduces XSA, a novel self attention variant that enhances context modeling by excluding self-information, demonstrating consistent improvements over standard SA.

Findings

XSA outperforms standard SA on language modeling tasks.

Performance gains increase with sequence length.

Improvements are consistent across models up to 2.7B parameters.

Abstract

We introduce exclusive self attention (XSA), a simple modification of self attention (SA) that improves Transformer's sequence modeling performance. The key idea is to constrain attention to capture only information orthogonal to the token's own value vector (thus excluding information of self position), encouraging better context modeling. Evaluated on the standard language modeling task, XSA consistently outperforms SA across model sizes up to 2.7B parameters and shows increasingly larger gains as sequence length grows.