How Focused Are LLMs? A Quantitative Study via Repetitive Deterministic Prediction Tasks

TL;DR

This study examines how large language models perform on repetitive deterministic tasks, revealing a sharp accuracy decline beyond a certain sequence length and proposing a physics-inspired model to explain this failure.

Contribution

The paper introduces a statistical physics-inspired model that explains the exponential decay in accuracy and the accuracy cliff in large language models on deterministic tasks.

Findings

Accuracy drops sharply beyond a characteristic length.

Models fail to perform independent operations reliably.

The proposed model captures the transition and explains internal interference.

Abstract

We investigate the performance of large language models on repetitive deterministic prediction tasks and study how the sequence accuracy rate scales with output length. Each such task involves repeating the same operation n times. Examples include letter replacement in strings following a given rule, integer addition, and multiplication of string operators in many body quantum mechanics. If the model performs the task through a simple repetition algorithm, the success rate should decay exponentially with sequence length. In contrast, our experiments on leading large language models reveal a sharp double exponential drop beyond a characteristic length scale, forming an accuracy cliff that marks the transition from reliable to unstable generation. This indicates that the models fail to execute each operation independently. To explain this phenomenon, we propose a statistical physics inspired model that captures the competition between external conditioning from the prompt and internal interference among generated tokens. The model quantitatively reproduces the observed crossover and provides an interpretable link between attention induced interference and sequence level failure. Fitting the model to empirical results across multiple models and tasks yields effective parameters that characterize the intrinsic error rate and error accumulation factor for each model task pair, offering a principled framework for understanding the limits of deterministic accuracy in large language models.