D-Models and E-Models: Diversity-Stability Trade-offs in the Sampling Behavior of Large Language Models

TL;DR

This paper investigates the sampling behaviors of large language models, revealing a dichotomy between models with high variability and those with stable probabilities, which impacts their suitability for different tasks.

Contribution

It introduces a classification of LLMs into D-models and E-models based on their sampling stability and alignment with task distributions, providing insights for model selection.

Findings

D-models exhibit high step-to-step variability in token probabilities.

E-models show more stable and task-aligned token probabilities.

Trade-offs between diversity and stability influence task performance.

Abstract

The predictive probability of the next token (P_token) in large language models (LLMs) is inextricably linked to the probability of relevance for the next piece of information, the purchase probability of the next product, and the execution probability of the next action-all of which fall under the scope of the task-level target distribution (P_task). While LLMs are known to generate samples that approximate real-world distributions, whether their fine-grained sampling probabilities faithfully align with task requirements remains an open question. Through controlled distribution-sampling simulations, we uncover a striking dichotomy in LLM behavior, distinguishing two model types: D-models (e.g. Qwen-2.5), whose P_token exhibits large step-to-step variability and poor alignment with P_task; and E-models (e.g. Mistral-Small), whose P_token is more stable and better aligned with P_task. We further evaluate these two model types in downstream tasks such as code generation and recommendation, revealing systematic trade-offs between diversity and stability that shape task outcomes. Finally, we analyze the internal properties of both model families to probe their underlying mechanisms. These findings offer foundational insights into the probabilistic sampling behavior of LLMs and provide practical guidance on when to favor D- versus E-models. For web-scale applications, including recommendation, search, and conversational agents, our results inform model selection and configuration to balance diversity with reliability under real-world uncertainty, providing a better level of interpretation.