The Position Curse: LLMs Struggle to Locate the Last Few Items in a List

TL;DR

Large language models struggle with retrieving the last few items in a list, especially backward, a failure termed the Position Curse, which affects code understanding and editing tasks.

Contribution

The paper characterizes the Position Curse in LLMs, introduces PosBench for position-focused training, and shows LoRA fine-tuning improves position retrieval but leaves room for improvement.

Findings

Backward retrieval lags forward retrieval across models.

LoRA fine-tuning improves position retrieval capabilities.

Absolute position retrieval performance remains far from saturation.

Abstract

Modern large language models (LLMs) can find a needle in a haystack (locating a single relevant fact buried among hundreds of thousands of irrelevant tokens) with near-saturated accuracy, yet fail to retrieve the last few items in a short list. We call this failure the Position Curse. For instance, even in a two-line code snippet, Claude Opus 4.6 misidentifies the second-to-last line most of the time. To characterize this failure, we evaluated two complementary queries: given a position in a sequence (of letters or words), retrieve the corresponding item; and given an item, return its position. Each position is specified as a forward or backward offset from an anchor, either an endpoint of the list (its start or end) or another item in the list. Across both open-source and frontier closed-source models, backward retrieval substantially lags forward retrieval. To test whether this capability can be rescued by post-training, we constructed PosBench, a position-focused training dataset. LoRA fine-tuning improves both forward and backward retrieval and generalizes to a held-out code-understanding benchmark (PyIndex), yet absolute performance remains far from saturated. As LLM coding agents increasingly operate over large codebases where precise indexing becomes essential for code understanding and editing, position-based retrieval emerges as a key capability for future pretraining objectives and model design.