ChatGPT's financial discrimination between rich and poor -- misaligned with human behavior and expectations

TL;DR

This paper investigates how ChatGPT's bargaining behavior in an ultimatum game differs from human expectations, revealing that ChatGPT favors rich receivers contrary to human beliefs and preferences, highlighting misalignment with human social norms.

Contribution

It introduces a novel experimental setup to analyze ChatGPT's behavior in bargaining scenarios and uncovers its preference for rich receivers, contrasting with human expectations.

Findings

Humans expect ChatGPT to favor poor receivers.

ChatGPT actually favors rich receivers.

Humans' beliefs about ChatGPT's preferences are incorrect.

Abstract

ChatGPT disrupted the application of machine-learning methods and drastically reduced the usage barrier. Chatbots are now widely used in a lot of different situations. They provide advice, assist in writing source code, or assess and summarize information from various sources. However, their scope is not only limited to aiding humans; they can also be used to take on tasks like negotiating or bargaining. To understand the implications of Chatbot usage on bargaining situations, we conduct a laboratory experiment with the ultimatum game. In the ultimatum game, two human players interact: The receiver decides on accepting or rejecting a monetary offer from the proposer. To shed light on the new bargaining situation, we let ChatGPT provide an offer to a human player. In the novel design, we vary the wealth of the receivers. Our results indicate that humans have the same beliefs about other humans and chatbots. However, our results contradict these beliefs in an important point: Humans favor poor receivers as correctly anticipated by the humans, but ChatGPT favors rich receivers which the humans did not expect to happen. These results imply that ChatGPT's answers are not aligned with those of humans and that humans do not anticipate this difference.