The Text Uncanny Valley: Non-Monotonic Performance Degradation in LLM Information Retrieval

TL;DR

This paper uncovers a non-linear degradation in LLM performance on corrupted text, revealing a 'Text Uncanny Valley' where detection accuracy dips then recovers as text fragmentation increases.

Contribution

It introduces the 'Text Uncanny Valley' phenomenon, explaining how word fragmentation causes a mode transition in LLMs, impacting their detection accuracy on imperfect text.

Findings

LLMs exhibit a U-shaped performance curve with increasing text corruption.

Regularization reduces the non-monotonic performance degradation.

The effect varies across models and tasks, diminishing with stronger models.

Abstract

Existing Large Language Model (LLM) benchmarks primarily focus on syntactically correct inputs, leaving a significant gap in evaluation on imperfect text. In this work, we study how word-boundary corruption affects how LLMs detect targeted information. By inserting whitespace characters within words to break them into fragments, LLMs' detection accuracy follows a U-shaped curve with the increase in insertion rate. We refer to this curve as the Text Uncanny Valley. To explain such observation, we propose a mode transition hypothesis: LLMs operate in a word-level mode for near-normal text and a character-level mode for heavily fragmented text, with the valley marking the disordered transition where neither mode is effective. Four experiments and one analysis are consistent with this account: in-context learning fails to rescue valley-bottom performance; regularizing the perturbation substantially reduces the U-shape; a math reasoning task replicates the U-shape for Gemini 3.0 Flash but not for stronger models, suggesting the effect is attenuated when tasks rely less on exact lexical alignment; and tokenization entropy peaks before the F1 minimum, consistent with a regime-conflict interpretation. These findings reveal a failure mode invisible to clean-text benchmarks yet directly relevant to any deployment scenario involving noisy or uncurated text inputs.