Predicting Biased Human Decision-Making with Large Language Models in Conversational Settings

TL;DR

This study investigates whether large language models can predict human decision biases in conversations, especially under cognitive load, and finds that GPT-4 models replicate human bias patterns more accurately than others.

Contribution

It demonstrates that LLMs, particularly GPT-4, can predict human biased decision-making and load-bias interactions in conversational settings, advancing AI's ability to simulate human cognition.

Findings

LLMs can predict human decision biases with reasonable accuracy.

Dialogue complexity increases cognitive load and bias effects.

GPT-4 models best replicate human bias patterns.

Abstract

We examine whether large language models (LLMs) can predict biased decision-making in conversational settings, and whether their predictions capture not only human cognitive biases but also how those effects change under cognitive load. In a pre-registered study (N = 1,648), participants completed six classic decision-making tasks via a chatbot with dialogues of varying complexity. Participants exhibited two well-documented cognitive biases: the Framing Effect and the Status Quo Bias. Increased dialogue complexity resulted in participants reporting higher mental demand. This increase in cognitive load selectively, but significantly, increased the effect of the biases, demonstrating the load-bias interaction. We then evaluated whether LLMs (GPT-4, GPT-5, and open-source models) could predict individual decisions given demographic information and prior dialogue. While results were mixed across choice problems, LLM predictions that incorporated dialogue context were significantly more accurate in several key scenarios. Importantly, their predictions reproduced the same bias patterns and load-bias interactions observed in humans. Across all models tested, the GPT-4 family consistently aligned with human behavior, outperforming GPT-5 and open-source models in both predictive accuracy and fidelity to human-like bias patterns. These findings advance our understanding of LLMs as tools for simulating human decision-making and inform the design of conversational agents that adapt to user biases.