Humans are more gullible than LLMs in believing common psychological myths

TL;DR

This study compares human and LLM belief in psychological myths, finding LLMs are less gullible, and demonstrates methods like retrieval-augmented generation to further reduce myth belief in LLMs.

Contribution

It introduces a systematic evaluation of myth belief in LLMs, showing how prompting strategies can mitigate misconceptions and revealing potential for debiasing in machine psychology.

Findings

LLMs have lower myth belief rates than humans.

Prompting strategies influence LLM responses.

Retrieval-augmented generation reduces myth belief.

Abstract

Despite widespread debunking, many psychological myths remain deeply entrenched. This paper investigates whether Large Language Models (LLMs) mimic human behaviour of myth belief and explores methods to mitigate such tendencies. Using 50 popular psychological myths, we evaluate myth belief across multiple LLMs under different prompting strategies, including retrieval-augmented generation and swaying prompts. Results show that LLMs exhibit significantly lower myth belief rates than humans, though user prompting can influence responses. RAG proves effective in reducing myth belief and reveals latent debiasing potential within LLMs. Our findings contribute to the emerging field of Machine Psychology and highlight how cognitive science methods can inform the evaluation and development of LLM-based systems.