Do large language models and humans have similar behaviors in causal inference with script knowledge?

TL;DR

This study compares human and large language model behaviors in causal inference within script-based stories, revealing that recent models partially mimic human responses but still struggle with integrating script knowledge.

Contribution

It provides a systematic comparison of human and LLM causal reasoning in script contexts, highlighting current models' limitations and partial alignment with human behavior.

Findings

Humans show longer reading times for causal conflicts.

Recent LLMs like GPT-3 correlate with human responses in some conditions.

All models fail to predict the lower surprise of no cause event.

Abstract

Recently, large pre-trained language models (LLMs) have demonstrated superior language understanding abilities, including zero-shot causal reasoning. However, it is unclear to what extent their capabilities are similar to human ones. We here study the processing of an event $B$ in a script-based story, which causally depends on a previous event $A$. In our manipulation, event $A$ is stated, negated, or omitted in an earlier section of the text. We first conducted a self-paced reading experiment, which showed that humans exhibit significantly longer reading times when causal conflicts exist ($\neg A \rightarrow B$) than under logical conditions ($A \rightarrow B$). However, reading times remain similar when cause A is not explicitly mentioned, indicating that humans can easily infer event B from their script knowledge. We then tested a variety of LLMs on the same data to check to what extent the models replicate human behavior. Our experiments show that 1) only recent LLMs, like GPT-3 or Vicuna, correlate with human behavior in the $\neg A \rightarrow B$ condition. 2) Despite this correlation, all models still fail to predict that $nil \rightarrow B$ is less surprising than $\neg A \rightarrow B$, indicating that LLMs still have difficulties integrating script knowledge. Our code and collected data set are available at https://github.com/tony-hong/causal-script.