Failure to Mix: Large language models struggle to answer according to desired probability distributions

TL;DR

This paper demonstrates that large language models fail to generate outputs following specified probability distributions, highlighting a significant limitation in their probabilistic exploration capabilities.

Contribution

The study systematically evaluates LLMs' ability to produce outputs according to target distributions, revealing a fundamental challenge in probabilistic output control.

Findings

LLMs tend to produce deterministic outputs despite probabilistic prompts

Requesting specific probability distributions results in near-deterministic responses

Modern LLMs do not accurately follow simple probabilistic distributions

Abstract

Scientific idea generation and selection requires exploration following a target probability distribution. In contrast, current AI benchmarks have objectively correct answers, and training large language models (LLMs) via reinforcement learning against these benchmarks discourages probabilistic exploration. Here, we conducted systematic experiments requesting LLMs to produce outputs following simple probabilistic distributions, and found that all modern LLMs tested grossly fail to follow the distributions. For example, requesting a binary output of "1" 49% of the time produces an answer of "0" nearly 100% of the time. This step function-like behavior of near-exclusively generating the output with marginally highest probability even overrules even strong in-built LLM biases.