Influence of Solution Efficiency and Valence of Instruction on Additive and Subtractive Solution Strategies in Humans and GPT-4

TL;DR

This study compares human and GPT-4 problem-solving strategies, revealing that GPT-4 exhibits stronger additive biases and that both human and AI strategies are influenced by task efficiency and instruction valence.

Contribution

It provides a comparative analysis of human and GPT-4 problem-solving behaviors across spatial and linguistic tasks, highlighting differences in biases and the influence of task instructions.

Findings

GPT-4 shows a stronger tendency towards additive strategies than humans.

Humans are less likely to use additive strategies when subtraction is more efficient.

GPT-4's use of additive strategies increases with positive instruction valence.

Abstract

Generative artificial intelligences, particularly large language models (LLMs), play an increasingly prominent role in human decision-making contexts, necessitating transparency about their capabilities. While prior studies have shown addition biases in humans (Adams et al., 2021) and OpenAI's GPT-3 (Winter et al., 2023), this study extends the research by comparing human and GPT-4 problem-solving across both spatial and linguistic tasks, with variations in solution efficiency and valence of task instruction. Four preregistered experiments with 588 participants from the U.S. and 680 GPT-4 iterations revealed a stronger tendency towards additive transformations in GPT-4 than in humans. Human participants were less likely to use additive strategies when subtraction was relatively more efficient than when addition and subtraction were equally efficient. GPT-4 exhibited the opposite behavior, with a strong addition bias when subtraction was more efficient. In terms of valence of task instruction, GPT-4's use of additive strategies increased when instructed to "improve" (positive) rather than "edit" (neutral). These findings demonstrate that biases in human problem-solving are amplified in GPT-4, and that LLM behavior differs from human efficiency-based strategies. This highlights the limitations of LLMs and the need for caution when using them in real-world applications.