Unveiling Attractor Cycles in Large Language Models: A Dynamical Systems View of Successive Paraphrasing

TL;DR

This paper applies dynamical systems theory to analyze large language models, revealing that successive paraphrasing often converges to stable cycles, which limits linguistic diversity and highlights inherent constraints in LLMs.

Contribution

It introduces a dynamical systems framework to study LLM behavior, demonstrating that iterative paraphrasing leads to attractor cycles, a novel perspective on LLM dynamics.

Findings

Successive paraphrasing converges to 2-cycle attractors.

LLMs tend to favor and amplify certain textual forms.

This convergence persists despite increased randomness or prompt variation.

Abstract

Dynamical systems theory provides a framework for analyzing iterative processes and evolution over time. Within such systems, repetitive transformations can lead to stable configurations, known as attractors, including fixed points and limit cycles. Applying this perspective to large language models (LLMs), which iteratively map input text to output text, provides a principled approach to characterizing long-term behaviors. Successive paraphrasing serves as a compelling testbed for exploring such dynamics, as paraphrases re-express the same underlying meaning with linguistic variation. Although LLMs are expected to explore a diverse set of paraphrases in the text space, our study reveals that successive paraphrasing converges to stable periodic states, such as 2-period attractor cycles, limiting linguistic diversity. This phenomenon is attributed to the self-reinforcing nature of LLMs, as they iteratively favour and amplify certain textual forms over others. This pattern persists with increasing generation randomness or alternating prompts and LLMs. These findings underscore inherent constraints in LLM generative capability, while offering a novel dynamical systems perspective for studying their expressive potential.