Layer-wise Positional Bias in Short-Context Language Modeling

TL;DR

This paper introduces an attribution-based framework to analyze how positional biases evolve across layers in short-context language models, revealing stable, architecture-specific importance profiles and biases like recency and primacy.

Contribution

It presents a novel layer-wise analysis method for positional biases in language models, uncovering stable importance profiles and their relation to model depth and architecture.

Findings

Recency bias increases with model depth.

Primacy bias diminishes through model depth.

Early layers favor content words over function words.

Abstract

Language models often show a preference for using information from specific positions in the input regardless of semantic relevance. While positional bias has been studied in various contexts, from attention sinks to task performance degradation in long-context settings, prior work has not established how these biases evolve across individual layers and input positions, or how they vary independent of task complexity. We introduce an attribution-based framework to analyze positional effects in short-context language modeling. Using layer conductance with a sliding-window approach, we quantify how each layer distributes importance across input positions, yielding layer-wise positional importance profiles. We find that these profiles are architecture-specific, stable across inputs, and invariant to lexical scrambling. Characterizing these profiles, we find prominent recency bias that increases with depth and subtle primacy bias that diminishes through model depth. Beyond positional structure, we also show that early layers preferentially weight content words over function words across all positions, while later layers lose this word-type differentiation.