Human-Like Intuitive Behavior and Reasoning Biases Emerged in Language Models -- and Disappeared in GPT-4

TL;DR

This paper examines how large language models initially exhibit human-like intuition and biases, but advanced models like GPT-4 have learned to avoid these errors, showing a progression towards more rational behavior.

Contribution

The study applies psychological tests to LLMs, revealing the emergence and subsequent disappearance of human-like biases as models become more advanced.

Findings

GPT-3 exhibits human-like intuition and biases.

GPT-4 and ChatGPT avoid these biases, showing rational decision-making.

Psychological testing reveals emergent traits in LLMs.

Abstract

Large language models (LLMs) are currently at the forefront of intertwining AI systems with human communication and everyday life. Therefore, it is of great importance to evaluate their emerging abilities. In this study, we show that LLMs, most notably GPT-3, exhibit behavior that strikingly resembles human-like intuition -- and the cognitive errors that come with it. However, LLMs with higher cognitive capabilities, in particular ChatGPT and GPT-4, learned to avoid succumbing to these errors and perform in a hyperrational manner. For our experiments, we probe LLMs with the Cognitive Reflection Test (CRT) as well as semantic illusions that were originally designed to investigate intuitive decision-making in humans. Moreover, we probe how sturdy the inclination for intuitive-like decision-making is. Our study demonstrates that investigating LLMs with methods from psychology has the potential to reveal otherwise unknown emergent traits.