Thinking Fast and Slow in Large Language Models

TL;DR

This paper explores how large language models exhibit human-like intuition and errors, but higher-capability models like GPT-4 show improved rationality, revealing emergent cognitive traits through psychological testing methods.

Contribution

It introduces a novel approach of applying psychological tests to analyze cognitive behaviors and errors in large language models, highlighting emergent traits in advanced models.

Findings

GPT-3 shows human-like intuition and errors.

GPT-4 and ChatGPT perform in a more rational manner.

Psychological testing reveals emergent cognitive traits in LLMs.

Abstract

Large language models (LLMs) are currently at the forefront of intertwining AI systems with human communication and everyday life. Therefore, it is of great importance to evaluate their emerging abilities. In this study, we show that LLMs like GPT-3 exhibit behavior that strikingly resembles human-like intuition - and the cognitive errors that come with it. However, LLMs with higher cognitive capabilities, in particular ChatGPT and GPT-4, learned to avoid succumbing to these errors and perform in a hyperrational manner. For our experiments, we probe LLMs with the Cognitive Reflection Test (CRT) as well as semantic illusions that were originally designed to investigate intuitive decision-making in humans. Our study demonstrates that investigating LLMs with methods from psychology has the potential to reveal otherwise unknown emergent traits.