Token Homogenization under Positional Bias

TL;DR

This paper explores how token representations in large language models tend to become uniform across layers, especially under positional bias, affecting model interpretability and performance.

Contribution

It provides empirical evidence linking token homogenization to positional bias and analyzes how attention mechanisms influence this phenomenon.

Findings

Tokens lose distinctiveness during processing.

Positional bias amplifies homogenization.

Homogenization depends on attention mechanisms.

Abstract

This paper investigates token homogenization - the convergence of token representations toward uniformity across transformer layers and its relationship to positional bias in large language models. We empirically examine whether homogenization occurs and how positional bias amplifies this effect. Through layer-wise similarity analysis and controlled experiments, we demonstrate that tokens systematically lose distinctiveness during processing, particularly when biased toward extremal positions. Our findings confirm both the existence of homogenization and its dependence on positional attention mechanisms.