Mind the (DH) Gap! A Contrast in Risky Choices Between Reasoning and Conversational LLMs

TL;DR

This paper investigates decision-making in large language models, revealing two categories—reasoning models and conversational models—with distinct behaviors under uncertainty, and highlights training for mathematical reasoning as a key differentiator.

Contribution

It classifies LLMs into two behavioral categories based on decision-making patterns and identifies training for mathematical reasoning as a crucial factor distinguishing them.

Findings

Reasoning models behave rationally and are insensitive to prospect framing.

Conversational models are less rational, more human-like, and sensitive to framing and order.

Training for mathematical reasoning differentiates reasoning models from conversational models.

Abstract

The use of large language models either as decision support systems, or in agentic workflows, is rapidly transforming the digital ecosystem. However, the understanding of LLM decision-making under uncertainty remains limited. We study LLM risky choices along two dimensions: (1) prospect representation (based on an explicit representation or outcome history) and (2) decision rationale (explanation). Our study, which involves 20 frontier and open LLMs, is complemented by a matched human subjects experiment, which provides one reference point, while an expected payoff maximizing rational agent model provides another. We find that LLMs cluster into two categories: reasoning models (RMs) and conversational models (CMs). RMs tend towards rational behavior, are insensitive to the order of prospects, gain/loss framing, and explanations, and behave similarly whether prospects are explicit or presented via a history of outcomes. CMs are significantly less rational, slightly more human-like, sensitive to prospect ordering, framing, and explanation, and exhibit a large description-history gap. Paired comparisons of open LLMs suggest that a key factor differentiating RMs and CMs is training for mathematical reasoning.