The Illusion of Stochasticity in LLMs

TL;DR

This paper reveals that Large Language Models struggle with reliable stochastic sampling, a key requirement for agentic behavior, exposing a fundamental flaw in their ability to generate true randomness from internal probability estimates.

Contribution

It provides a comprehensive empirical analysis demonstrating the limitations of LLMs in converting internal probabilities into stochastic outputs, highlighting a core failure in their sampling capabilities.

Findings

Powerful models can convert random seeds to target distributions

LLMs fail to sample directly from specific distributions reliably

Sampling from internal probability estimates is fundamentally flawed in LLMs

Abstract

In this work, we demonstrate that reliable stochastic sampling is a fundamental yet unfulfilled requirement for Large Language Models (LLMs) operating as agents. Agentic systems are frequently required to sample from distributions, often inferred from observed data, a process which needs to be emulated by the LLM. This leads to a distinct failure point: while standard RL agents rely on external sampling mechanisms, LLMs fail to map their internal probability estimates to their stochastic outputs. Through rigorous empirical analysis across multiple model families, model sizes, prompting styles, and distributions, we demonstrate the extent of this failure. Crucially, we show that while powerful frontier models can convert provided random seeds to target distributions, their ability to sample directly from specific distributions is fundamentally flawed.