LLM economicus? Mapping the Behavioral Biases of LLMs via Utility Theory

TL;DR

This paper uses utility theory to evaluate and compare the economic biases of various large language models, revealing their inconsistent and non-human-like decision-making behaviors.

Contribution

It introduces a novel utility theory-based framework to quantify and analyze economic biases in LLMs, bridging economic theory and AI behavior assessment.

Findings

LLMs exhibit mixed economic and human-like biases.

Most LLMs lack consistent economic behavior across different settings.

Prompting can influence the economic biases of LLMs.

Abstract

Humans are not homo economicus (i.e., rational economic beings). As humans, we exhibit systematic behavioral biases such as loss aversion, anchoring, framing, etc., which lead us to make suboptimal economic decisions. Insofar as such biases may be embedded in text data on which large language models (LLMs) are trained, to what extent are LLMs prone to the same behavioral biases? Understanding these biases in LLMs is crucial for deploying LLMs to support human decision-making. We propose utility theory-a paradigm at the core of modern economic theory-as an approach to evaluate the economic biases of LLMs. Utility theory enables the quantification and comparison of economic behavior against benchmarks such as perfect rationality or human behavior. To demonstrate our approach, we quantify and compare the economic behavior of a variety of open- and closed-source LLMs. We find that the economic behavior of current LLMs is neither entirely human-like nor entirely economicus-like. We also find that most current LLMs struggle to maintain consistent economic behavior across settings. Finally, we illustrate how our approach can measure the effect of interventions such as prompting on economic biases.