Is Anisotropy Inherent to Transformers?

TL;DR

This paper investigates whether anisotropy in Transformer models is an inherent property or a consequence of training, showing it occurs across modalities and objectives, suggesting it may be fundamental to Transformer architectures.

Contribution

The study provides empirical evidence that anisotropy is intrinsic to Transformers, independent of training objectives or data distributions, across multiple modalities.

Findings

Anisotropy occurs in language models with different objectives.

Anisotropy is observed in Transformers trained on various modalities.

It may be an inherent characteristic of Transformer architectures.

Abstract

The representation degeneration problem is a phenomenon that is widely observed among self-supervised learning methods based on Transformers. In NLP, it takes the form of anisotropy, a singular property of hidden representations which makes them unexpectedly close to each other in terms of angular distance (cosine-similarity). Some recent works tend to show that anisotropy is a consequence of optimizing the cross-entropy loss on long-tailed distributions of tokens. We show in this paper that anisotropy can also be observed empirically in language models with specific objectives that should not suffer directly from the same consequences. We also show that the anisotropy problem extends to Transformers trained on other modalities. Our observations tend to demonstrate that anisotropy might actually be inherent to Transformers-based models.