RePo: Language Models with Context Re-Positioning

TL;DR

RePo introduces a differentiable context re-positioning mechanism for language models, reducing cognitive load and improving performance on complex tasks involving long and noisy contexts by better capturing contextual dependencies.

Contribution

RePo presents a novel, learnable position assignment module that enhances language models' ability to handle complex, noisy, and long contexts, outperforming fixed positional schemes.

Findings

Improved performance on tasks with noisy and structured data

Better attention allocation to relevant distant information

Maintains competitive results on short-context tasks

Abstract

In-context learning is fundamental to modern Large Language Models (LLMs); however, prevailing architectures impose a rigid and fixed contextual structure by assigning linear or constant positional indices. Drawing on Cognitive Load Theory (CLT), we argue that this uninformative structure increases extraneous cognitive load, consuming finite working memory capacity that should be allocated to deep reasoning and attention allocation. To address this, we propose RePo, a novel mechanism that reduces extraneous load via context re-positioning. Unlike standard approaches, RePo utilizes a differentiable module, $f_\phi$, to assign token positions that capture contextual dependencies, rather than replying on pre-defined order. By continually pre-training on the OLMo-2 1B & 7B models, we demonstrate that RePo consistently enhances performance on tasks involving noisy contexts, structured data, and longer context length, while maintaining competitive performance on general short-context tasks. Detailed analysis reveals that RePo successfully allocate higher attention to distant but relevant information, assign positions in dense and non-linear space, and capture the intrinsic structure of the input context. We will open-source the code and model weights. Our code is at https://github.com/SakanaAI/repo.