The First Drop of Ink: Nonlinear Impact of Misleading Information in Long-Context Reasoning

TL;DR

This paper uncovers a nonlinear pattern where a small increase in misleading information sharply degrades large language models' long-context reasoning, emphasizing the importance of retrieval accuracy.

Contribution

It introduces the 'First Drop of Ink' effect, demonstrating how minimal distractors disproportionately impact performance and analyzing underlying attention mechanics.

Findings

Performance drops sharply with initial small increases in distractors.

Filtering mainly improves performance by reducing context length.

Near-zero distractor proportion is needed for substantial recovery.

Abstract

As large language models are increasingly deployed in retrieval-augmented generation and agentic systems that accumulate extensive context, understanding how distracting information affects long-context performance becomes critical. Prior work shows that semantically relevant yet misleading documents degrade performance, but the quantitative relationship between the proportion of distractors and performance remains unstudied. In this work, we systematically vary the hard-distractor proportion in fixed-length contexts, revealing a striking nonlinear pattern: as the proportion of hard distractors increases, performance drops sharply within the first small fraction, while the remainder of the range yields only marginal additional decline. We term this ''The First Drop of Ink'' effect, analogous to how a single drop of ink contaminates water. Our theoretical and empirical analyses grounded in attention mechanics show that hard distractors capture disproportionate attention even at small proportions, with diminishing marginal impact as their proportion grows. Controlled experiments further show that filtering gains mainly come from context-length reduction rather than distractor removal; substantial recovery requires reducing the hard-distractor proportion to near zero, highlighting the importance of upstream retrieval precision.