Implicit Causality-biases in humans and LLMs as a tool for benchmarking LLM discourse capabilities

TL;DR

This study compares human and multilingual LLM discourse biases related to implicit causality verbs, developing a benchmark to evaluate LLMs' discourse understanding by analyzing coreference, coherence, and referring expression biases.

Contribution

It introduces a benchmark for assessing LLM discourse capabilities using well-established human discourse biases as a proxy.

Findings

Largest monolingual LLM shows human-like coreference bias

No LLM displayed typical human coherence explanation bias

All LLMs preferred subject arguments with simpler referring expressions

Abstract

In this paper, we compare data generated with mono- and multilingual LLMs spanning a range of model sizes with data provided by human participants in an experimental setting investigating well-established discourse biases. Beyond the comparison as such, we aim to develop a benchmark to assess the capabilities of LLMs with discourse biases as a robust proxy for more general discourse understanding capabilities. More specifically, we investigated Implicit Causality verbs, for which psycholinguistic research has found participants to display biases with regard to three phenomena:\ the establishment of (i) coreference relations (Experiment 1), (ii) coherence relations (Experiment 2), and (iii) the use of particular referring expressions (Experiments 3 and 4). With regard to coreference biases we found only the largest monolingual LLM (German Bloom 6.4B) to display more human-like biases. For coherence relation, no LLM displayed the explanation bias usually found for humans. For referring expressions, all LLMs displayed a preference for referring to subject arguments with simpler forms than to objects. However, no bias effect on referring expression was found, as opposed to recent studies investigating human biases.