Hidden in the Haystack: Smaller Needles are More Difficult for LLMs to Find

TL;DR

This study systematically investigates how the size of relevant context affects large language models' ability to perform long-context question answering across diverse domains, revealing that smaller contexts significantly hinder performance.

Contribution

It is the first comprehensive analysis of gold context size impact on LLMs, demonstrating its critical role independent of other factors across multiple benchmarks and models.

Findings

Smaller gold contexts lead to lower model accuracy.

Performance degradation is consistent across different domains and models.

Gold context size is an independent predictor of success.

Abstract

Large language models (LLMs) face significant challenges with needle-in-ahaystack tasks, where relevant information ("the needle") must be drawn from a large pool of irrelevant context ("the haystack"). Previous studies have highlighted positional bias and distractor quantity as critical factors affecting model performance, yet the influence of gold context size, the length of the answer-containing document, has received little attention. We present the first systematic study of gold context size in long-context question answering, spanning three diverse benchmarks (general knowledge, biomedical reasoning, and mathematical reasoning), eleven state-of-the-art LLMs (including recent reasoning models), and more than 150K controlled runs. Our experiments reveal that LLM performance drops sharply when the gold context is shorter, i.e., smaller gold contexts consistently degrade model performance and amplify positional sensitivity, posing a major challenge for agentic systems that must integrate scattered, fine-grained information of varying lengths. This effect persists under rigorous confounder analysis: even after controlling for gold context position, answer token repetition, gold-to-distractor ratio, distractor volume, and domain specificity, gold context size remains a decisive, independent predictor of success. Our work provides clear insights to guide the design of robust, context-aware LLM-driven systems.