Markovian Generation Chains in Large Language Models

TL;DR

This paper models the iterative text generation process of large language models as Markov chains, analyzing how repeated inference affects sentence diversity and convergence, with implications for multi-agent LLM systems.

Contribution

It introduces a Markovian framework for understanding iterative LLM outputs and analyzes how parameters influence diversity and convergence.

Findings

Iterative process can lead to convergence or ongoing novelty.

Temperature and initial input significantly affect diversity.

Markov chain analysis provides insights into LLM dynamics.

Abstract

The widespread use of large language models (LLMs) raises an important question: how do texts evolve when they are repeatedly processed by LLMs? In this paper, we define this iterative inference process as Markovian generation chains, where each step takes a specific prompt template and the previous output as input, without including any prior memory. In iterative rephrasing and round-trip translation experiments, the output either converges to a small recurrent set or continues to produce novel sentences over a finite horizon. Through sentence-level Markov chain modeling and analysis of simulated data, we show that iterative process can either increase or reduce sentence diversity depending on factors such as the temperature parameter and the initial input sentence. These results offer valuable insights into the dynamics of iterative LLM inference and their implications for multi-agent LLM systems.