Do Large Language Models Show Biases in Causal Learning?

TL;DR

This study investigates whether large language models exhibit biases in causal learning, revealing that they often display causal illusions similar to humans, especially in scenarios with spurious correlations or misleading cues.

Contribution

The paper provides the first systematic analysis of causal bias in LLMs, demonstrating their tendencies to infer causality erroneously in structured causal inference tasks.

Findings

LLMs show causal illusion biases comparable to humans in open-ended tasks.

Models exhibit higher bias in null-contingency and temporal cue scenarios.

LLMs do not reliably internalize normative principles of causal reasoning.

Abstract

Causal learning is the cognitive process of developing the capability of making causal inferences based on available information, often guided by normative principles. This process is prone to errors and biases, such as the illusion of causality, in which people perceive a causal relationship between two variables despite lacking supporting evidence. This cognitive bias has been proposed to underlie many societal problems, including social prejudice, stereotype formation, misinformation, and superstitious thinking. In this research, we investigate whether large language models (LLMs) develop causal illusions, both in real-world and controlled laboratory contexts of causal learning and inference. To this end, we built a dataset of over 2K samples including purely correlational cases, situations with null contingency, and cases where temporal information excludes the possibility of causality by placing the potential effect before the cause. We then prompted the models to make statements or answer causal questions to evaluate their tendencies to infer causation erroneously in these structured settings. Our findings show a strong presence of causal illusion bias in LLMs. Specifically, in open-ended generation tasks involving spurious correlations, the models displayed bias at levels comparable to, or even lower than, those observed in similar studies on human subjects. However, when faced with null-contingency scenarios or temporal cues that negate causal relationships, where it was required to respond on a 0-100 scale, the models exhibited significantly higher bias. These findings suggest that the models have not uniformly, consistently, or reliably internalized the normative principles essential for accurate causal learning.