John praised Mary because he? Implicit Causality Bias and Its Interaction with Explicit Cues in LMs

TL;DR

This paper examines whether pre-trained language models encode implicit causality biases from interpersonal verbs and how they handle explicit versus implicit causal cues, revealing a tendency to prioritize lexical cues over sentence-level semantics.

Contribution

It demonstrates that PLMs encode implicit causality biases to varying degrees and tend to rely more on lexical cues than on explicit sentence-level information during inference.

Findings

PLMs encode implicit causality biases with varying strength.

Explicit causes in subordinate clauses can cause processing delays.

PLMs prioritize lexical cues over higher-order sentence semantics.

Abstract

Some interpersonal verbs can implicitly attribute causality to either their subject or their object and are therefore said to carry an implicit causality (IC) bias. Through this bias, causal links can be inferred from a narrative, aiding language comprehension. We investigate whether pre-trained language models (PLMs) encode IC bias and use it at inference time. We find that to be the case, albeit to different degrees, for three distinct PLM architectures. However, causes do not always need to be implicit -- when a cause is explicitly stated in a subordinate clause, an incongruent IC bias associated with the verb in the main clause leads to a delay in human processing. We hypothesize that the temporary challenge humans face in integrating the two contradicting signals, one from the lexical semantics of the verb, one from the sentence-level semantics, would be reflected in higher error rates for models on tasks dependent on causal links. The results of our study lend support to this hypothesis, suggesting that PLMs tend to prioritize lexical patterns over higher-order signals.