Rethinking Prospect Theory for LLMs: Revealing the Instability of Decision-Making under Epistemic Uncertainty

TL;DR

This paper critically examines the applicability and robustness of Prospect Theory in modeling LLM decision-making under linguistic epistemic uncertainty, revealing significant instability and limitations.

Contribution

It introduces a three-stage workflow to evaluate PT parameters in LLMs and demonstrates PT's instability under epistemic markers, challenging its suitability for LLM decision modeling.

Findings

PT modeling in LLMs is inconsistent across different models.

PT is not robust to epistemic uncertainty in LLMs.

Applying PT to LLMs under linguistic uncertainty is unreliable.

Abstract

Prospect Theory (PT) models human decision-making behaviour under uncertainty, among which linguistic uncertainty is commonly adopted in real-world scenarios. Although recent studies have developed some frameworks to test PT parameters for Large Language Models (LLMs), few have considered the fitness of PT itself on LLMs. Moreover, whether PT is robust under linguistic uncertainty perturbations, especially epistemic markers (e.g. "likely"), remains highly under-explored. To address these gaps, we design a three-stage workflow based on a classic behavioural economics experimental setup. We first estimate PT parameters with economics questions and evaluate PT's fitness with performance metrics. We then derive probability mappings for epistemic markers in the same context, and inject these mappings into the prompt to investigate the stability of PT parameters. Our findings suggest that modelling LLMs' decision-making with PT is not consistently reliable across models, and applying Prospect Theory to LLMs is likely not robust to epistemic uncertainty. The findings caution against the deployment of PT-based frameworks in real-world applications where epistemic ambiguity is prevalent, giving valuable insights in behaviour interpretation and future alignment direction for LLM decision-making.