Experimental Pragmatics with Machines: Testing LLM Predictions for the Inferences of Plain and Embedded Disjunctions

TL;DR

This study evaluates how well large language models predict human-like inferences from plain and embedded disjunctions, revealing that models largely mirror human reasoning patterns in these linguistic inferences.

Contribution

It introduces a novel experimental approach comparing LLM predictions with human inferences for disjunctions and implicatures, highlighting the models' ability to replicate human reasoning.

Findings

Models align with humans in inference patterns

Significant differences observed between disjunctions and implicatures

Models capture fine-grained distinctions in inferences

Abstract

Human communication is based on a variety of inferences that we draw from sentences, often going beyond what is literally said. While there is wide agreement on the basic distinction between entailment, implicature, and presupposition, the status of many inferences remains controversial. In this paper, we focus on three inferences of plain and embedded disjunctions, and compare them with regular scalar implicatures. We investigate this comparison from the novel perspective of the predictions of state-of-the-art large language models, using the same experimental paradigms as recent studies investigating the same inferences with humans. The results of our best performing models mostly align with those of humans, both in the large differences we find between those inferences and implicatures, as well as in fine-grained distinctions among different aspects of those inferences.