Assessing Large Language Models' ability to predict how humans balance self-interest and the interest of others

TL;DR

This study evaluates GPT-4's ability to predict human decision-making in social dilemmas, revealing its strengths in capturing behavioral patterns but also its biases in estimating self-interest and altruism.

Contribution

It is the first to compare advanced chatbots' predictions of human social behavior across diverse experiments, highlighting GPT-4's unique capabilities and biases.

Findings

GPT-4 correctly identifies behavioral classes

GPT-4 underestimates self-interest and inequity-aversion

GPT-4 overestimates altruistic behavior

Abstract

Generative artificial intelligence (AI) holds enormous potential to revolutionize decision-making processes, from everyday to high-stake scenarios. By leveraging generative AI, humans can benefit from data-driven insights and predictions, enhancing their ability to make informed decisions that consider a wide array of factors and potential outcomes. However, as many decisions carry social implications, for AI to be a reliable assistant for decision-making it is crucial that it is able to capture the balance between self-interest and the interest of others. We investigate the ability of three of the most advanced chatbots to predict dictator game decisions across 108 experiments with human participants from 12 countries. We find that only GPT-4 (not Bard nor Bing) correctly captures qualitative behavioral patterns, identifying three major classes of behavior: self-interested, inequity-averse, and fully altruistic. Nonetheless, GPT-4 consistently underestimates self-interest and inequity-aversion, while overestimating altruistic behavior. This bias has significant implications for AI developers and users, as overly optimistic expectations about human altruism may lead to disappointment, frustration, suboptimal decisions in public policy or business contexts, and even social conflict.